Nov 03 20:00:27.887882 ip-10-0-143-48 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 20:00:28.263857 ip-10-0-143-48 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:00:28.263857 ip-10-0-143-48 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 20:00:28.263857 ip-10-0-143-48 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:00:28.263857 ip-10-0-143-48 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:00:28.267718 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267635    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:00:28.267718 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267717    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:00:28.267718 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267722    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:00:28.267819 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267725    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:00:28.267819 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267728    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:00:28.267819 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267731    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:00:28.267819 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267733    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:00:28.267819 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267736    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:00:28.267819 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267738    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:00:28.267819 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267741    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:00:28.267819 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267744    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:00:28.267819 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267747    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:00:28.267819 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267749    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:00:28.267819 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267752    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:00:28.267819 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267754    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:00:28.267819 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267757    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:00:28.267819 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267760    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:00:28.267819 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267762    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:00:28.267819 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267765    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:00:28.267819 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267767    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:00:28.267819 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267771    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:00:28.267819 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267774    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:00:28.268259 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267776    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:00:28.268259 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267779    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:00:28.268259 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267781    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:00:28.268259 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267784    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:00:28.268259 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267786    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:00:28.268259 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267789    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:00:28.268259 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267791    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:00:28.268259 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267794    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:00:28.268259 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267796    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:00:28.268259 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267799    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:00:28.268259 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267801    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:00:28.268259 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267803    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:00:28.268259 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267807    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:00:28.268259 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267810    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:00:28.268259 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267812    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:00:28.268259 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267815    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:00:28.268259 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267818    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:00:28.268259 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267821    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:00:28.268259 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267824    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:00:28.268758 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267827    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:00:28.268758 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267830    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:00:28.268758 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267833    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:00:28.268758 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267835    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:00:28.268758 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267838    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:00:28.268758 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267840    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:00:28.268758 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267843    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:00:28.268758 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267845    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:00:28.268758 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267848    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:00:28.268758 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267850    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:00:28.268758 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267853    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:00:28.268758 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267856    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:00:28.268758 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267859    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:00:28.268758 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267861    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:00:28.268758 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267863    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:00:28.268758 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267866    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:00:28.268758 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267868    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:00:28.268758 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267871    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:00:28.268758 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267873    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:00:28.268758 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267875    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:00:28.269221 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267878    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:00:28.269221 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267880    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:00:28.269221 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267882    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:00:28.269221 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267886    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:00:28.269221 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267888    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:00:28.269221 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267892    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:00:28.269221 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267894    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:00:28.269221 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267897    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:00:28.269221 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267900    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:00:28.269221 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267902    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:00:28.269221 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267906    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:00:28.269221 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267910    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:00:28.269221 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267912    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:00:28.269221 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267917    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:00:28.269221 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267921    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:00:28.269221 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267924    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:00:28.269221 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267928    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:00:28.269221 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267932    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:00:28.269221 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267935    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:00:28.269692 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267938    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:00:28.269692 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267940    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:00:28.269692 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267943    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:00:28.269692 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267946    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:00:28.269692 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267948    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:00:28.269692 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267951    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:00:28.269692 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267953    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:00:28.269692 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267956    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:00:28.269692 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267959    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:00:28.269692 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267962    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:00:28.269692 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267964    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:00:28.269692 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267967    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:00:28.269692 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267969    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:00:28.269692 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267972    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:00:28.269692 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267976    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:00:28.269692 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267978    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:00:28.269692 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267981    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:00:28.269692 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267983    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:00:28.269692 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267986    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:00:28.269692 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267988    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:00:28.270148 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267992    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:00:28.270148 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267994    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:00:28.270148 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267997    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:00:28.270148 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.267999    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:00:28.270148 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268002    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:00:28.270148 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268004    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:00:28.270148 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268816    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:00:28.270148 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268824    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:00:28.270148 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268828    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:00:28.270148 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268831    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:00:28.270148 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268834    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:00:28.270148 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268836    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:00:28.270148 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268839    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:00:28.270148 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268842    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:00:28.270148 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268844    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:00:28.270148 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268847    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:00:28.270148 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268849    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:00:28.270148 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268851    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:00:28.270148 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268854    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:00:28.270608 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268856    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:00:28.270608 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268859    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:00:28.270608 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268862    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:00:28.270608 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268865    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:00:28.270608 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268868    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:00:28.270608 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268870    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:00:28.270608 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268873    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:00:28.270608 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268875    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:00:28.270608 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268878    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:00:28.270608 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268880    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:00:28.270608 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268883    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:00:28.270608 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268886    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:00:28.270608 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268889    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:00:28.270608 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268892    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:00:28.270608 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268894    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:00:28.270608 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268897    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:00:28.270608 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268899    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:00:28.270608 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268901    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:00:28.270608 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268904    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:00:28.271108 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268907    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:00:28.271108 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268909    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:00:28.271108 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268912    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:00:28.271108 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268914    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:00:28.271108 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268917    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:00:28.271108 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268919    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:00:28.271108 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268922    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:00:28.271108 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268924    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:00:28.271108 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268926    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:00:28.271108 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268930    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:00:28.271108 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268934    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:00:28.271108 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268937    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:00:28.271108 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268940    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:00:28.271108 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268942    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:00:28.271108 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268946    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:00:28.271108 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268948    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:00:28.271108 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268951    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:00:28.271108 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268955    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:00:28.271528 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268959    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:00:28.271528 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268963    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:00:28.271528 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268966    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:00:28.271528 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268969    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:00:28.271528 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268972    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:00:28.271528 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268974    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:00:28.271528 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268977    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:00:28.271528 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268980    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:00:28.271528 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268983    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:00:28.271528 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268985    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:00:28.271528 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268987    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:00:28.271528 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268990    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:00:28.271528 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268993    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:00:28.271528 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268995    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:00:28.271528 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.268997    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:00:28.271528 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269000    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:00:28.271528 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269003    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:00:28.271528 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269006    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:00:28.271528 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269008    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:00:28.271528 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269011    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:00:28.271528 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269013    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:00:28.272028 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269015    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:00:28.272028 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269018    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:00:28.272028 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269020    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:00:28.272028 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269023    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:00:28.272028 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269025    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:00:28.272028 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269028    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:00:28.272028 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269030    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:00:28.272028 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269033    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:00:28.272028 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269035    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:00:28.272028 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269038    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:00:28.272028 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269040    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:00:28.272028 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269047    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:00:28.272028 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269049    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:00:28.272028 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269052    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:00:28.272028 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269055    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:00:28.272028 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269057    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:00:28.272028 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269059    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:00:28.272028 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269062    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:00:28.272028 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269065    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:00:28.272028 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269067    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:00:28.272504 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269069    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:00:28.272504 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269072    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:00:28.272504 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269074    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:00:28.272504 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269076    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:00:28.272504 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269079    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:00:28.272504 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269081    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:00:28.272504 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269084    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:00:28.272504 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269087    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:00:28.272504 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269090    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:00:28.272504 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269092    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:00:28.272504 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269095    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:00:28.272504 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269097    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:00:28.272504 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269100    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:00:28.272504 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269102    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:00:28.272504 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269106    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:00:28.272504 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269590    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:00:28.272504 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269610    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:00:28.272504 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269613    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:00:28.272504 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269616    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:00:28.272504 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269619    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:00:28.272993 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269621    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:00:28.272993 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269624    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:00:28.272993 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269626    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:00:28.272993 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269631    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:00:28.272993 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269634    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:00:28.272993 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269638    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:00:28.272993 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269642    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:00:28.272993 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269646    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:00:28.272993 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269649    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:00:28.272993 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269652    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:00:28.272993 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269655    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:00:28.272993 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269657    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:00:28.272993 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269660    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:00:28.272993 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269662    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:00:28.272993 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269665    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:00:28.272993 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269667    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:00:28.272993 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269670    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:00:28.272993 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269672    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:00:28.272993 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269675    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:00:28.273442 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269680    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:00:28.273442 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269684    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:00:28.273442 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269688    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:00:28.273442 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269691    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:00:28.273442 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269695    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:00:28.273442 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269697    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:00:28.273442 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269700    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:00:28.273442 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269702    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:00:28.273442 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269705    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:00:28.273442 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269708    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:00:28.273442 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269710    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:00:28.273442 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269713    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:00:28.273442 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269715    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:00:28.273442 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269718    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:00:28.273442 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269720    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:00:28.273442 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269723    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:00:28.273442 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269728    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:00:28.273442 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269730    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:00:28.273442 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269733    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:00:28.273442 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269735    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:00:28.273916 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269738    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:00:28.273916 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269740    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:00:28.273916 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269743    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:00:28.273916 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269745    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:00:28.273916 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269747    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:00:28.273916 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269750    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:00:28.273916 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269752    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:00:28.273916 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269755    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:00:28.273916 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269757    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:00:28.273916 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269759    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:00:28.273916 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269762    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:00:28.273916 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269764    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:00:28.273916 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269767    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:00:28.273916 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269770    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:00:28.273916 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269772    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:00:28.273916 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269775    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:00:28.273916 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269777    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:00:28.273916 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269780    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:00:28.273916 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269782    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:00:28.273916 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269785    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:00:28.274379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269787    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:00:28.274379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269790    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:00:28.274379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269792    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:00:28.274379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269795    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:00:28.274379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269798    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:00:28.274379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269800    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:00:28.274379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269803    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:00:28.274379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269805    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:00:28.274379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269809    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:00:28.274379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269811    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:00:28.274379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269814    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:00:28.274379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269817    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:00:28.274379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269820    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:00:28.274379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269823    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:00:28.274379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269825    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:00:28.274379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269828    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:00:28.274379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269830    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:00:28.274379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269833    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:00:28.274379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269836    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:00:28.274863 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269838    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:00:28.274863 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269840    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:00:28.274863 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269843    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:00:28.274863 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269845    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:00:28.274863 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269848    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:00:28.274863 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269850    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:00:28.274863 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269853    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:00:28.274863 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269855    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:00:28.274863 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269858    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:00:28.274863 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269860    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:00:28.274863 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269863    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:00:28.274863 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269865    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:00:28.274863 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269868    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:00:28.274863 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269870    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:00:28.274863 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269873    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:00:28.274863 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269875    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:00:28.274863 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269878    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:00:28.274863 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269880    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:00:28.274863 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269883    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:00:28.274863 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269885    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:00:28.275417 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269888    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:00:28.275417 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269892    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:00:28.275417 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.269894    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:00:28.275417 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270817    2575 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 20:00:28.275417 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270827    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 20:00:28.275417 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270833    2575 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 20:00:28.275417 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270838    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 20:00:28.275417 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270842    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 20:00:28.275417 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270845    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 20:00:28.275417 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270849    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 20:00:28.275417 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270854    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 20:00:28.275417 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270857    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 20:00:28.275417 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270860    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 20:00:28.275417 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270863    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 20:00:28.275417 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270867    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 20:00:28.275417 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270870    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 20:00:28.275417 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270873    2575 flags.go:64] FLAG: --cgroup-root=""
Nov 03 20:00:28.275417 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270875    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 20:00:28.275417 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270879    2575 flags.go:64] FLAG: --client-ca-file=""
Nov 03 20:00:28.275417 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270881    2575 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 20:00:28.275417 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270885    2575 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 20:00:28.275417 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270889    2575 flags.go:64] FLAG: --cluster-domain=""
Nov 03 20:00:28.275417 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270892    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 20:00:28.275942 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270895    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 20:00:28.275942 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270899    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 20:00:28.275942 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270902    2575 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 20:00:28.275942 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270906    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 20:00:28.275942 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270909    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 20:00:28.275942 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270914    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 20:00:28.275942 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270918    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 20:00:28.275942 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270921    2575 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 20:00:28.275942 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270924    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 20:00:28.275942 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270927    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 20:00:28.275942 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270930    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 20:00:28.275942 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270933    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 20:00:28.275942 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270937    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 20:00:28.275942 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270940    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 20:00:28.275942 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270943    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 20:00:28.275942 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270946    2575 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 20:00:28.275942 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270949    2575 flags.go:64] FLAG: --enable-server="true"
Nov 03 20:00:28.275942 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270952    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 20:00:28.275942 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270955    2575 flags.go:64] FLAG: --event-burst="100"
Nov 03 20:00:28.275942 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270958    2575 flags.go:64] FLAG: --event-qps="50"
Nov 03 20:00:28.275942 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270961    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 20:00:28.275942 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270964    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 20:00:28.275942 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270967    2575 flags.go:64] FLAG: --eviction-hard=""
Nov 03 20:00:28.275942 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270971    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 20:00:28.276574 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270973    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 20:00:28.276574 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270976    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 20:00:28.276574 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270979    2575 flags.go:64] FLAG: --eviction-soft=""
Nov 03 20:00:28.276574 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270982    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 20:00:28.276574 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270985    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 20:00:28.276574 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270988    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 20:00:28.276574 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270991    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 20:00:28.276574 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270994    2575 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 20:00:28.276574 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270997    2575 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 20:00:28.276574 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.270999    2575 flags.go:64] FLAG: --feature-gates=""
Nov 03 20:00:28.276574 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271003    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 20:00:28.276574 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271006    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 20:00:28.276574 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271009    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 20:00:28.276574 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271014    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 20:00:28.276574 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271017    2575 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 20:00:28.276574 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271020    2575 flags.go:64] FLAG: --help="false"
Nov 03 20:00:28.276574 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271024    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:00:28.276574 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271027    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 20:00:28.276574 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271030    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 20:00:28.276574 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271033    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 20:00:28.276574 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271036    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 20:00:28.276574 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271039    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 20:00:28.276574 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271042    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 20:00:28.277135 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271045    2575 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 20:00:28.277135 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271048    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 20:00:28.277135 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271051    2575 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 20:00:28.277135 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271054    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 20:00:28.277135 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271057    2575 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 20:00:28.277135 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271060    2575 flags.go:64] FLAG: --kube-reserved=""
Nov 03 20:00:28.277135 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271063    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 20:00:28.277135 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271066    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 20:00:28.277135 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271069    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 20:00:28.277135 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271071    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 20:00:28.277135 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271074    2575 flags.go:64] FLAG: --lock-file=""
Nov 03 20:00:28.277135 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271077    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 20:00:28.277135 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271079    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 20:00:28.277135 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271083    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 20:00:28.277135 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271088    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 20:00:28.277135 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271091    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 20:00:28.277135 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271095    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 20:00:28.277135 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271098    2575 flags.go:64] FLAG: --logging-format="text"
Nov 03 20:00:28.277135 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271101    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 20:00:28.277135 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271104    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 20:00:28.277135 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271106    2575 flags.go:64] FLAG: --manifest-url=""
Nov 03 20:00:28.277135 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271109    2575 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 20:00:28.277135 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271115    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 20:00:28.277135 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271118    2575 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 20:00:28.277135 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271122    2575 flags.go:64] FLAG: --max-pods="110"
Nov 03 20:00:28.277923 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271125    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 20:00:28.277923 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271128    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 20:00:28.277923 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271130    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 20:00:28.277923 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271133    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 20:00:28.277923 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271136    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 20:00:28.277923 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271139    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 20:00:28.277923 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271142    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 20:00:28.277923 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271149    2575 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 20:00:28.277923 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271152    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 20:00:28.277923 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271155    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 20:00:28.277923 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271158    2575 flags.go:64] FLAG: --pod-cidr=""
Nov 03 20:00:28.277923 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271161    2575 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 20:00:28.277923 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271164    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 20:00:28.277923 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271167    2575 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 20:00:28.277923 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271170    2575 flags.go:64] FLAG: --port="10250"
Nov 03 20:00:28.277923 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271173    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 20:00:28.277923 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271175    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0ae94d7e1e01fa8f2"
Nov 03 20:00:28.277923 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271178    2575 flags.go:64] FLAG: --qos-reserved=""
Nov 03 20:00:28.277923 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271181    2575 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 20:00:28.277923 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271184    2575 flags.go:64] FLAG: --register-node="true"
Nov 03 20:00:28.277923 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271203    2575 flags.go:64] FLAG: --register-with-taints=""
Nov 03 20:00:28.277923 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271208    2575 flags.go:64] FLAG: --registry-burst="10"
Nov 03 20:00:28.277923 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271211    2575 flags.go:64] FLAG: --registry-qps="5"
Nov 03 20:00:28.277923 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271213    2575 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 20:00:28.277923 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271217    2575 flags.go:64] FLAG: --reserved-memory=""
Nov 03 20:00:28.277923 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271221    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 20:00:28.279022 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271223    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 20:00:28.279022 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271226    2575 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 20:00:28.279022 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271229    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 20:00:28.279022 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271233    2575 flags.go:64] FLAG: --runonce="false"
Nov 03 20:00:28.279022 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271235    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 20:00:28.279022 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271238    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 20:00:28.279022 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271241    2575 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 20:00:28.279022 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271244    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 20:00:28.279022 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271247    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 20:00:28.279022 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271250    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 20:00:28.279022 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271253    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 20:00:28.279022 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271256    2575 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 20:00:28.279022 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271259    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 20:00:28.279022 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271261    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 20:00:28.279022 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271264    2575 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 20:00:28.279022 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271267    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 20:00:28.279022 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271270    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 20:00:28.279022 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271273    2575 flags.go:64] FLAG: --system-cgroups=""
Nov 03 20:00:28.279022 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271276    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 20:00:28.279022 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271281    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 20:00:28.279022 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271283    2575 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 20:00:28.279022 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271286    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 20:00:28.279022 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271290    2575 flags.go:64] FLAG: --tls-min-version=""
Nov 03 20:00:28.279022 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271292    2575 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 20:00:28.279022 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271295    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 20:00:28.280127 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271298    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 20:00:28.280127 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271301    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 20:00:28.280127 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271304    2575 flags.go:64] FLAG: --v="2"
Nov 03 20:00:28.280127 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271308    2575 flags.go:64] FLAG: --version="false"
Nov 03 20:00:28.280127 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271312    2575 flags.go:64] FLAG: --vmodule=""
Nov 03 20:00:28.280127 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271316    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 20:00:28.280127 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271320    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 20:00:28.280127 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271453    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:00:28.280127 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271457    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:00:28.280127 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271460    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:00:28.280127 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271463    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:00:28.280127 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271465    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:00:28.280127 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271467    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:00:28.280127 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271471    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:00:28.280127 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271474    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:00:28.280127 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271476    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:00:28.280127 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271478    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:00:28.280127 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271481    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:00:28.280127 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271483    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:00:28.280127 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271486    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:00:28.280127 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271488    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:00:28.280785 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271491    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:00:28.280785 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271493    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:00:28.280785 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271496    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:00:28.280785 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271498    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:00:28.280785 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271500    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:00:28.280785 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271503    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:00:28.280785 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271505    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:00:28.280785 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271508    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:00:28.280785 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271510    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:00:28.280785 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271512    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:00:28.280785 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271516    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:00:28.280785 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271520    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:00:28.280785 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271524    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:00:28.280785 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271526    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:00:28.280785 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271530    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:00:28.280785 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271533    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:00:28.280785 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271536    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:00:28.280785 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271538    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:00:28.280785 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271542    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:00:28.281266 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271544    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:00:28.281266 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271547    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:00:28.281266 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271550    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:00:28.281266 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271552    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:00:28.281266 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271554    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:00:28.281266 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271558    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:00:28.281266 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271560    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:00:28.281266 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271563    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:00:28.281266 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271565    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:00:28.281266 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271568    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:00:28.281266 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271571    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:00:28.281266 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271574    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:00:28.281266 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271578    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:00:28.281266 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271581    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:00:28.281266 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271583    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:00:28.281266 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271586    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:00:28.281266 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271588    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:00:28.281266 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271591    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:00:28.281266 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271607    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:00:28.281266 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271610    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:00:28.281744 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271612    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:00:28.281744 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271615    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:00:28.281744 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271617    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:00:28.281744 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271620    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:00:28.281744 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271622    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:00:28.281744 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271625    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:00:28.281744 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271627    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:00:28.281744 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271630    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:00:28.281744 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271632    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:00:28.281744 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271634    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:00:28.281744 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271637    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:00:28.281744 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271639    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:00:28.281744 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271642    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:00:28.281744 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271645    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:00:28.281744 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271647    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:00:28.281744 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271650    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:00:28.281744 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271652    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:00:28.281744 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271656    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:00:28.281744 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271658    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:00:28.282184 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271660    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:00:28.282184 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271663    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:00:28.282184 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271665    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:00:28.282184 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271668    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:00:28.282184 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271671    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:00:28.282184 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271673    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:00:28.282184 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271675    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:00:28.282184 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271678    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:00:28.282184 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271680    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:00:28.282184 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271683    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:00:28.282184 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271685    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:00:28.282184 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271687    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:00:28.282184 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271690    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:00:28.282184 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271693    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:00:28.282184 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271695    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:00:28.282184 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271699    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:00:28.282184 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271701    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:00:28.282184 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271704    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:00:28.282184 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271706    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:00:28.282184 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271709    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:00:28.282666 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271711    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:00:28.282666 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271714    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:00:28.282666 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271716    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:00:28.282666 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271718    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:00:28.282666 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271721    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:00:28.282666 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271723    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:00:28.282666 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271726    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:00:28.282666 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271728    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:00:28.282666 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271731    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:00:28.282666 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271733    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:00:28.282666 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271737    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:00:28.282666 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271739    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:00:28.282666 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271742    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:00:28.282666 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271744    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:00:28.282666 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.271770    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:00:28.282666 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271807    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:00:28.283057 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271811    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:00:28.283057 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271813    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:00:28.283057 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271816    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:00:28.283057 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271819    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:00:28.283057 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271821    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:00:28.283057 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271824    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:00:28.283057 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271826    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:00:28.283057 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271829    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:00:28.283057 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271831    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:00:28.283057 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271834    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:00:28.283057 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271836    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:00:28.283057 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271840    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:00:28.283057 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271843    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:00:28.283057 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271845    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:00:28.283057 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271848    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:00:28.283057 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271850    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:00:28.283057 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271852    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:00:28.283057 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271855    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:00:28.283057 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271857    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:00:28.283495 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271860    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:00:28.283495 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271862    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:00:28.283495 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271866    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:00:28.283495 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271869    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:00:28.283495 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271873    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:00:28.283495 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271875    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:00:28.283495 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271878    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:00:28.283495 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271881    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:00:28.283495 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271884    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:00:28.283495 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271886    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:00:28.283495 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271889    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:00:28.283495 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271892    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:00:28.283495 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271894    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:00:28.283495 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271896    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:00:28.283495 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271899    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:00:28.283495 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271902    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:00:28.283495 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271904    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:00:28.283495 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271906    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:00:28.283495 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271909    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:00:28.283954 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271911    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:00:28.283954 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271914    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:00:28.283954 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271917    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:00:28.283954 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271921    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:00:28.283954 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271924    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:00:28.283954 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271929    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:00:28.283954 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271932    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:00:28.283954 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271934    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:00:28.283954 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271937    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:00:28.283954 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271939    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:00:28.283954 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271942    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:00:28.283954 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271944    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:00:28.283954 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271946    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:00:28.283954 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271949    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:00:28.283954 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271951    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:00:28.283954 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271954    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:00:28.283954 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271957    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:00:28.283954 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271960    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:00:28.283954 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271963    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:00:28.284388 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271965    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:00:28.284388 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271968    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:00:28.284388 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271972    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:00:28.284388 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271974    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:00:28.284388 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271977    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:00:28.284388 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271979    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:00:28.284388 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271982    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:00:28.284388 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271984    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:00:28.284388 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271987    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:00:28.284388 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271989    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:00:28.284388 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271991    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:00:28.284388 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271994    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:00:28.284388 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271996    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:00:28.284388 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.271999    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:00:28.284388 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272001    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:00:28.284388 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272004    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:00:28.284388 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272006    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:00:28.284388 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272009    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:00:28.284388 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272013    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:00:28.284388 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272016    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:00:28.284893 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272019    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:00:28.284893 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272021    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:00:28.284893 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272023    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:00:28.284893 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272026    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:00:28.284893 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272028    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:00:28.284893 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272031    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:00:28.284893 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272034    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:00:28.284893 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272036    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:00:28.284893 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272039    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:00:28.284893 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272042    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:00:28.284893 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272044    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:00:28.284893 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272047    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:00:28.284893 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272050    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:00:28.284893 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272052    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:00:28.284893 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272055    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:00:28.284893 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272058    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:00:28.284893 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272060    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:00:28.284893 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272062    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:00:28.284893 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272065    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:00:28.284893 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272067    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:00:28.285362 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272070    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:00:28.285362 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272072    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:00:28.285362 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272074    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:00:28.285362 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272077    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:00:28.285362 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272079    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:00:28.285362 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272082    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:00:28.285362 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272085    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:00:28.285362 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.272087    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:00:28.285362 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.281291    2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 20:00:28.285362 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.281307    2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 20:00:28.285362 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.281328    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 20:00:28.285362 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.281333    2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 20:00:28.285362 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281391    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:00:28.285362 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281395    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:00:28.285362 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281398    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:00:28.285362 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281401    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:00:28.285362 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281404    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:00:28.285362 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281406    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:00:28.285362 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281409    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:00:28.285362 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281412    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:00:28.285858 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281414    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:00:28.285858 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281417    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:00:28.285858 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281419    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:00:28.285858 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281422    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:00:28.285858 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281425    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:00:28.285858 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281427    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:00:28.285858 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281430    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:00:28.285858 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281433    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:00:28.285858 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281435    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:00:28.285858 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281438    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:00:28.285858 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281440    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:00:28.285858 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281443    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:00:28.285858 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281445    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:00:28.285858 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281448    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:00:28.285858 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281452    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:00:28.285858 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281456    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:00:28.285858 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281459    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:00:28.285858 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281462    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:00:28.285858 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281464    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:00:28.286309 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281471    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:00:28.286309 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281474    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:00:28.286309 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281477    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:00:28.286309 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281480    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:00:28.286309 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281482    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:00:28.286309 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281485    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:00:28.286309 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281487    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:00:28.286309 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281490    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:00:28.286309 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281493    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:00:28.286309 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281496    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:00:28.286309 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281499    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:00:28.286309 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281501    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:00:28.286309 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281504    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:00:28.286309 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281506    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:00:28.286309 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281509    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:00:28.286309 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281511    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:00:28.286309 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281514    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:00:28.286309 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281516    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:00:28.286309 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281519    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:00:28.286839 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281522    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:00:28.286839 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281525    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:00:28.286839 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281528    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:00:28.286839 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281530    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:00:28.286839 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281533    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:00:28.286839 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281536    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:00:28.286839 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281538    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:00:28.286839 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281540    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:00:28.286839 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281543    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:00:28.286839 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281546    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:00:28.286839 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281548    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:00:28.286839 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281550    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:00:28.286839 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281553    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:00:28.286839 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281555    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:00:28.286839 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281557    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:00:28.286839 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281560    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:00:28.286839 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281562    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:00:28.286839 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281565    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:00:28.286839 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281567    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:00:28.287271 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281570    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:00:28.287271 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281572    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:00:28.287271 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281574    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:00:28.287271 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281577    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:00:28.287271 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281579    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:00:28.287271 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281582    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:00:28.287271 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281584    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:00:28.287271 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281587    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:00:28.287271 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281589    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:00:28.287271 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281592    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:00:28.287271 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281610    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:00:28.287271 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281614    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:00:28.287271 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281617    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:00:28.287271 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281619    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:00:28.287271 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281623    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:00:28.287271 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281626    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:00:28.287271 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281628    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:00:28.287271 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281631    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:00:28.287271 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281633    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:00:28.287271 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281635    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:00:28.287809 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281638    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:00:28.287809 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281641    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:00:28.287809 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281643    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:00:28.287809 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281646    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:00:28.287809 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281648    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:00:28.287809 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281651    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:00:28.287809 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281653    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:00:28.287809 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281656    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:00:28.287809 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281658    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:00:28.287809 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281661    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:00:28.287809 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281663    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:00:28.287809 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281666    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:00:28.287809 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281668    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:00:28.287809 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281670    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:00:28.287809 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281673    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:00:28.287809 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281675    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:00:28.287809 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281678    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:00:28.287809 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281680    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:00:28.287809 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281684    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:00:28.288270 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281687    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:00:28.288270 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281690    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:00:28.288270 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.281720    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:00:28.288270 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281860    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:00:28.288270 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281866    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:00:28.288270 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281868    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:00:28.288270 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281871    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:00:28.288270 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281874    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:00:28.288270 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281877    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:00:28.288270 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281880    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:00:28.288270 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281883    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:00:28.288270 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281886    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:00:28.288270 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281888    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:00:28.288270 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281890    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:00:28.288270 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281894    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:00:28.288270 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281898    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:00:28.288687 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281901    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:00:28.288687 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281903    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:00:28.288687 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281906    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:00:28.288687 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281908    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:00:28.288687 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281911    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:00:28.288687 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281914    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:00:28.288687 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281916    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:00:28.288687 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281919    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:00:28.288687 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281921    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:00:28.288687 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281924    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:00:28.288687 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281926    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:00:28.288687 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281929    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:00:28.288687 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281932    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:00:28.288687 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281935    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:00:28.288687 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281937    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:00:28.288687 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281940    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:00:28.288687 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281943    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:00:28.288687 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281946    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:00:28.288687 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281948    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:00:28.289130 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281952    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:00:28.289130 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281955    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:00:28.289130 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281958    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:00:28.289130 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281962    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:00:28.289130 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281964    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:00:28.289130 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281967    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:00:28.289130 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281970    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:00:28.289130 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281972    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:00:28.289130 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281975    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:00:28.289130 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281977    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:00:28.289130 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281979    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:00:28.289130 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281982    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:00:28.289130 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281984    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:00:28.289130 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281987    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:00:28.289130 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281989    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:00:28.289130 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281992    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:00:28.289130 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281994    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:00:28.289130 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281997    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:00:28.289130 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.281999    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:00:28.289571 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282002    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:00:28.289571 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282004    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:00:28.289571 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282006    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:00:28.289571 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282009    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:00:28.289571 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282011    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:00:28.289571 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282014    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:00:28.289571 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282016    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:00:28.289571 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282019    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:00:28.289571 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282021    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:00:28.289571 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282024    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:00:28.289571 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282026    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:00:28.289571 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282029    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:00:28.289571 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282031    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:00:28.289571 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282033    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:00:28.289571 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282036    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:00:28.289571 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282038    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:00:28.289571 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282041    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:00:28.289571 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282043    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:00:28.289571 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282046    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:00:28.290056 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282049    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:00:28.290056 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282052    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:00:28.290056 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282055    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:00:28.290056 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282057    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:00:28.290056 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282059    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:00:28.290056 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282062    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:00:28.290056 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282064    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:00:28.290056 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282067    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:00:28.290056 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282069    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:00:28.290056 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282071    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:00:28.290056 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282074    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:00:28.290056 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282077    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:00:28.290056 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282079    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:00:28.290056 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282081    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:00:28.290056 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282084    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:00:28.290056 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282086    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:00:28.290056 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282089    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:00:28.290056 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282091    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:00:28.290056 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282094    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:00:28.290056 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282096    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:00:28.290505 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282098    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:00:28.290505 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282101    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:00:28.290505 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282105    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:00:28.290505 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282108    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:00:28.290505 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282110    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:00:28.290505 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282113    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:00:28.290505 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282115    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:00:28.290505 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282118    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:00:28.290505 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282120    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:00:28.290505 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282123    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:00:28.290505 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282125    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:00:28.290505 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282127    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:00:28.290505 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282130    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:00:28.290505 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282132    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:00:28.290505 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282135    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:00:28.290505 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282138    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:00:28.291005 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.282160    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:00:28.291005 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282192    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:00:28.291005 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282196    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:00:28.291005 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282199    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:00:28.291005 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282202    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:00:28.291005 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282205    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:00:28.291005 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282208    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:00:28.291005 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282211    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:00:28.291005 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282213    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:00:28.291005 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282215    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:00:28.291005 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282218    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:00:28.291005 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282220    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:00:28.291005 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282223    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:00:28.291005 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282225    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:00:28.291005 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282227    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:00:28.291005 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282230    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:00:28.291403 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282232    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:00:28.291403 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282235    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:00:28.291403 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282238    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:00:28.291403 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282242    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:00:28.291403 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282244    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:00:28.291403 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282246    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:00:28.291403 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282249    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:00:28.291403 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282252    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:00:28.291403 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282254    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:00:28.291403 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282257    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:00:28.291403 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282259    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:00:28.291403 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282261    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:00:28.291403 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282264    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:00:28.291403 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282266    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:00:28.291403 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282269    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:00:28.291403 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282271    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:00:28.291403 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282274    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:00:28.291403 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282277    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:00:28.291403 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282279    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:00:28.291403 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282282    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:00:28.291890 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282284    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:00:28.291890 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282288    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:00:28.291890 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282291    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:00:28.291890 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282295    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:00:28.291890 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282297    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:00:28.291890 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282300    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:00:28.291890 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282302    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:00:28.291890 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282305    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:00:28.291890 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282307    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:00:28.291890 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282310    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:00:28.291890 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282312    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:00:28.291890 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282315    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:00:28.291890 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282317    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:00:28.291890 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282320    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:00:28.291890 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282322    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:00:28.291890 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282326    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:00:28.291890 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282328    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:00:28.291890 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282331    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:00:28.292311 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282334    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:00:28.292311 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282336    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:00:28.292311 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282339    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:00:28.292311 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282341    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:00:28.292311 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282345    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:00:28.292311 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282349    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:00:28.292311 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282352    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:00:28.292311 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282354    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:00:28.292311 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282356    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:00:28.292311 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282359    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:00:28.292311 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282362    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:00:28.292311 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282365    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:00:28.292311 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282367    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:00:28.292311 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282369    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:00:28.292311 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282372    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:00:28.292311 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282374    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:00:28.292311 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282377    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:00:28.292311 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282380    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:00:28.292311 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282383    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:00:28.292311 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282385    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:00:28.292905 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282388    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:00:28.292905 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282390    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:00:28.292905 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282392    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:00:28.292905 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282395    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:00:28.292905 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282397    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:00:28.292905 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282399    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:00:28.292905 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282402    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:00:28.292905 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282404    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:00:28.292905 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282407    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:00:28.292905 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282411    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:00:28.292905 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282414    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:00:28.292905 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282417    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:00:28.292905 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282420    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:00:28.292905 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282422    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:00:28.292905 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282425    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:00:28.292905 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282427    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:00:28.292905 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282430    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:00:28.292905 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282432    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:00:28.292905 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282434    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:00:28.293379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282437    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:00:28.293379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282439    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:00:28.293379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282442    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:00:28.293379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282445    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:00:28.293379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282448    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:00:28.293379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282451    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:00:28.293379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282453    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:00:28.293379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282456    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:00:28.293379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282458    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:00:28.293379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282461    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:00:28.293379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282465    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:00:28.293379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282468    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:00:28.293379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282470    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:00:28.293379 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:28.282472    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:00:28.293379 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.282526    2575 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 20:00:28.293379 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.285516    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 20:00:28.293379 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.286413    2575 server.go:1014] "Starting client certificate rotation"
Nov 03 20:00:28.293379 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.286503    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 20:00:28.293379 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.287155    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 20:00:28.293848 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.288265    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:00:28.293848 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.292622    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:00:28.306818 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.306803    2575 log.go:25] "Validated CRI v1 runtime API"
Nov 03 20:00:28.310653 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.310638    2575 log.go:25] "Validated CRI v1 image API"
Nov 03 20:00:28.311590 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.311566    2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 20:00:28.314122 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.314098    2575 fs.go:135] Filesystem UUIDs: map[318f595b-ab71-426b-90bf-36ab2c4d05d7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e36ff02b-8f16-4764-8530-928f047586f2:/dev/nvme0n1p4]
Nov 03 20:00:28.314186 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.314121    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 20:00:28.320141 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.320118    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 20:00:28.321268 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.321155    2575 manager.go:217] Machine: {Timestamp:2026-11-03 20:00:28.319236312 +0000 UTC m=+0.312392701 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108703 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b5f5b51f5c786985616f94002379c SystemUUID:ec2b5f5b-51f5-c786-9856-16f94002379c BootID:fe805aa2-b88f-417f-8a4c-154126457afa Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e5:b6:35:8d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e5:b6:35:8d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:59:ad:98:cb:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 20:00:28.321268 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.321267    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 20:00:28.321392 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.321380    2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 20:00:28.323802 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.323780    2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 20:00:28.323964 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.323804    2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-48.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 20:00:28.324011 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.323968    2575 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 20:00:28.324011 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.323976    2575 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 20:00:28.324011 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.324006    2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:00:28.324085 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.324025    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:00:28.324768 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.324758    2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 20:00:28.328705 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.328694    2575 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 20:00:28.328824 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.328815    2575 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 20:00:28.328853 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.328847    2575 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 20:00:28.328881 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.328863    2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 20:00:28.328926 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.328880    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 20:00:28.328926 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.328894    2575 kubelet.go:405] "Adding apiserver pod source"
Nov 03 20:00:28.328926 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.328905    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 20:00:28.329277 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.329266    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:00:28.331290 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.331268    2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 20:00:28.335718 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.335703    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:00:28.336467 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.336455    2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 20:00:28.336508 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.336486    2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 20:00:28.336744 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.336728    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 20:00:28.336775 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.336754    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 20:00:28.336775 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.336762    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 20:00:28.336775 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.336768    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 20:00:28.336775 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.336774    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 20:00:28.336873 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.336780    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 20:00:28.337236 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.337227    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 20:00:28.337267 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.337236    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 20:00:28.337267 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.337245    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 20:00:28.337267 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.337251    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 20:00:28.337267 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.337260    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 20:00:28.337267 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.337268    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 20:00:28.338139 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.338128    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 20:00:28.338186 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.338142    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 20:00:28.341869 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.341854    2575 server.go:1263] "Started kubelet"
Nov 03 20:00:28.342046 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.341994    2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 20:00:28.342306 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.342110    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 20:00:28.342420 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.342369    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 20:00:28.342568 ip-10-0-143-48 systemd[1]: Started Kubernetes Kubelet.
Nov 03 20:00:28.343085 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.343063    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-48.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 20:00:28.343840 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.343811    2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 20:00:28.344755 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.344739    2575 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 20:00:28.345640 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:28.345613    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-48.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 20:00:28.345716 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:28.345627    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 20:00:28.347155 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.347136    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pp69v"
Nov 03 20:00:28.347572 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:28.347554    2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 20:00:28.349902 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.349881    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 20:00:28.351002 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.350986    2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 20:00:28.351113 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:28.351094    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-48.us-east-2.compute.internal\" not found"
Nov 03 20:00:28.351634 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.351617    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 20:00:28.352075 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.351956    2575 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 20:00:28.352075 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.351969    2575 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 20:00:28.352279 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.352121    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 20:00:28.352279 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.352135    2575 factory.go:55] Registering systemd factory
Nov 03 20:00:28.352279 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.352142    2575 factory.go:223] Registration of the systemd container factory successfully
Nov 03 20:00:28.352279 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.352231    2575 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 20:00:28.352279 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.352239    2575 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 20:00:28.352515 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:28.351288    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-48.us-east-2.compute.internal.18e4a16a0622659a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-48.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-48.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 20:00:28.341822874 +0000 UTC m=+0.334979264,LastTimestamp:2026-11-03 20:00:28.341822874 +0000 UTC m=+0.334979264,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-48.us-east-2.compute.internal,}"
Nov 03 20:00:28.352756 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.352728    2575 factory.go:153] Registering CRI-O factory
Nov 03 20:00:28.352756 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.352746    2575 factory.go:223] Registration of the crio container factory successfully
Nov 03 20:00:28.352874 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.352763    2575 factory.go:103] Registering Raw factory
Nov 03 20:00:28.352874 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.352787    2575 manager.go:1199] Started watching for new ooms in manager
Nov 03 20:00:28.353786 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.353770    2575 manager.go:319] Starting recovery of all containers
Nov 03 20:00:28.358550 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:28.358524    2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-48.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 20:00:28.360822 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.360797    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 20:00:28.364235 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.364213    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pp69v"
Nov 03 20:00:28.364569 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:28.364543    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 20:00:28.367752 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.367734    2575 manager.go:324] Recovery completed
Nov 03 20:00:28.371568 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.371555    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:00:28.373838 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.373820    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-48.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:00:28.373913 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.373855    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-48.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:00:28.373913 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.373870    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-48.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:00:28.374402 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.374389    2575 cpu_manager.go:226] "Starting" policy="none"
Nov 03 20:00:28.374465 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.374403    2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 20:00:28.374465 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.374392    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:00:28.374465 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.374422    2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 20:00:28.375071 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.375055    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-48.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:00:28.375152 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.375085    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-48.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:00:28.375152 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.375099    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-48.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:00:28.376243 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:28.376169    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-48.us-east-2.compute.internal.18e4a16a080ae688  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-48.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-48.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-48.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 20:00:28.373837448 +0000 UTC m=+0.366993844,LastTimestamp:2026-11-03 20:00:28.373837448 +0000 UTC m=+0.366993844,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-48.us-east-2.compute.internal,}"
Nov 03 20:00:28.376489 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.376476    2575 policy_none.go:50] "Start"
Nov 03 20:00:28.376534 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.376494    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 20:00:28.376534 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.376530    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 20:00:28.378636 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.378623    2575 policy_none.go:44] "Start"
Nov 03 20:00:28.417313 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.417171    2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 20:00:28.421315 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.421300    2575 manager.go:341] "Starting Device Plugin manager"
Nov 03 20:00:28.421387 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:28.421331    2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 20:00:28.421387 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.421339    2575 server.go:87] "Starting device plugin registration server"
Nov 03 20:00:28.421555 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.421542    2575 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 20:00:28.421621 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.421555    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 20:00:28.421672 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.421642    2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 20:00:28.421716 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.421709    2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 20:00:28.421754 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.421717    2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 20:00:28.422230 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:28.422211    2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 20:00:28.422299 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:28.422260    2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-48.us-east-2.compute.internal\" not found"
Nov 03 20:00:28.516824 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.516770    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 20:00:28.516824 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.516796    2575 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 20:00:28.516824 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.516815    2575 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 20:00:28.518903 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:28.516848    2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 20:00:28.520772 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.520758    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:00:28.521832 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.521820    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:00:28.522587 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.522570    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-48.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:00:28.522699 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.522625    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-48.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:00:28.522699 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.522637    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-48.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:00:28.522699 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.522665    2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:00:28.531313 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.531298    2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:00:28.531381 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:28.531322    2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-48.us-east-2.compute.internal\": node \"ip-10-0-143-48.us-east-2.compute.internal\" not found"
Nov 03 20:00:28.548568 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:28.548550    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-48.us-east-2.compute.internal\" not found"
Nov 03 20:00:28.617743 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.617715    2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal"]
Nov 03 20:00:28.617858 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.617772    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:00:28.618618 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.618584    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-48.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:00:28.618691 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.618635    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-48.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:00:28.618691 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.618649    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-48.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:00:28.620767 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.620755    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:00:28.620901 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.620888    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:00:28.620938 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.620918    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:00:28.621589 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.621570    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-48.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:00:28.621709 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.621614    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-48.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:00:28.621709 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.621628    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-48.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:00:28.621709 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.621631    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-48.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:00:28.621709 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.621659    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-48.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:00:28.621709 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.621671    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-48.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:00:28.623636 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.623620    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:00:28.623693 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.623646    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:00:28.624356 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.624341    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-48.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:00:28.624441 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.624371    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-48.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:00:28.624441 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.624385    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-48.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:00:28.648917 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:28.648898    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-48.us-east-2.compute.internal\" not found"
Nov 03 20:00:28.654013 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:28.653998    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-48.us-east-2.compute.internal\" not found" node="ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:00:28.654583 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.654563    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b45f8e0dedc76d41e693c990769490b8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal\" (UID: \"b45f8e0dedc76d41e693c990769490b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:00:28.654648 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.654614    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc3d49b24d21c77c008662d39f9bb541-config\") pod \"kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal\" (UID: \"cc3d49b24d21c77c008662d39f9bb541\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:00:28.654688 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.654662    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b45f8e0dedc76d41e693c990769490b8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal\" (UID: \"b45f8e0dedc76d41e693c990769490b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:00:28.656699 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:28.656683    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-48.us-east-2.compute.internal\" not found" node="ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:00:28.749936 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:28.749906    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-48.us-east-2.compute.internal\" not found"
Nov 03 20:00:28.755269 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.755240    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b45f8e0dedc76d41e693c990769490b8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal\" (UID: \"b45f8e0dedc76d41e693c990769490b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:00:28.755356 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.755278    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b45f8e0dedc76d41e693c990769490b8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal\" (UID: \"b45f8e0dedc76d41e693c990769490b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:00:28.755356 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.755309    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc3d49b24d21c77c008662d39f9bb541-config\") pod \"kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal\" (UID: \"cc3d49b24d21c77c008662d39f9bb541\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:00:28.755356 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.755333    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b45f8e0dedc76d41e693c990769490b8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal\" (UID: \"b45f8e0dedc76d41e693c990769490b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:00:28.755356 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.755351    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b45f8e0dedc76d41e693c990769490b8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal\" (UID: \"b45f8e0dedc76d41e693c990769490b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:00:28.755498 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.755337    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc3d49b24d21c77c008662d39f9bb541-config\") pod \"kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal\" (UID: \"cc3d49b24d21c77c008662d39f9bb541\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:00:28.850749 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:28.850651    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-48.us-east-2.compute.internal\" not found"
Nov 03 20:00:28.951121 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:28.951091    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-48.us-east-2.compute.internal\" not found"
Nov 03 20:00:28.957431 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.957410    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:00:28.960444 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:28.960432    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:00:29.051926 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:29.051897    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-48.us-east-2.compute.internal\" not found"
Nov 03 20:00:29.152342 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:29.152257    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-48.us-east-2.compute.internal\" not found"
Nov 03 20:00:29.252753 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:29.252725    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-48.us-east-2.compute.internal\" not found"
Nov 03 20:00:29.286136 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:29.286112    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 20:00:29.286577 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:29.286245    2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 20:00:29.317284 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:29.317258    2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:00:29.353534 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:29.353504    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-48.us-east-2.compute.internal\" not found"
Nov 03 20:00:29.370192 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:29.370129    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:55:29 +0000 UTC" deadline="2028-05-01 10:10:34.371296754 +0000 UTC"
Nov 03 20:00:29.370192 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:29.370191    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13070h10m5.001109112s"
Nov 03 20:00:29.453646 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:29.453579    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-48.us-east-2.compute.internal\" not found"
Nov 03 20:00:29.531953 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:29.531919    2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:00:29.554242 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:29.554220    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-48.us-east-2.compute.internal\" not found"
Nov 03 20:00:29.617257 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:29.617223    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc3d49b24d21c77c008662d39f9bb541.slice/crio-072b4d3337117c5ea0ce179d4b73bad04b5790a500582f1468d6595e2fed9a06 WatchSource:0}: Error finding container 072b4d3337117c5ea0ce179d4b73bad04b5790a500582f1468d6595e2fed9a06: Status 404 returned error can't find the container with id 072b4d3337117c5ea0ce179d4b73bad04b5790a500582f1468d6595e2fed9a06
Nov 03 20:00:29.620677 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:29.620653    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb45f8e0dedc76d41e693c990769490b8.slice/crio-f47a37dac525b92bc09e460b56f9cd458f1074daf1136c74e3f6247f5b025d21 WatchSource:0}: Error finding container f47a37dac525b92bc09e460b56f9cd458f1074daf1136c74e3f6247f5b025d21: Status 404 returned error can't find the container with id f47a37dac525b92bc09e460b56f9cd458f1074daf1136c74e3f6247f5b025d21
Nov 03 20:00:29.623807 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:29.623793    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:00:29.654961 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:29.654943    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-48.us-east-2.compute.internal\" not found"
Nov 03 20:00:29.755485 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:29.755431    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-48.us-east-2.compute.internal\" not found"
Nov 03 20:00:29.792198 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:29.792176    2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:00:29.851393 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:29.851373    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:00:29.863644 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:29.863630    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:00:29.864971 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:29.864959    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:00:29.875807 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:29.875792    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:00:30.330330 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.329993    2575 apiserver.go:52] "Watching apiserver"
Nov 03 20:00:30.338860 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.338032    2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 20:00:30.338860 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.338445    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-w72hd","openshift-image-registry/node-ca-tt4df","openshift-network-diagnostics/network-check-target-h6crz","openshift-multus/multus-additional-cni-plugins-86vz2","openshift-multus/multus-mq5xz","openshift-multus/network-metrics-daemon-286gh","openshift-network-operator/iptables-alerter-v9qll","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb","openshift-dns/node-resolver-g2rp5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-lcp7r","kube-system/global-pull-secret-syncer-x8wrn","kube-system/konnectivity-agent-k5b5s","kube-system/kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal"]
Nov 03 20:00:30.339910 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.339889    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:00:30.341333 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.341313    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v9qll"
Nov 03 20:00:30.344305 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.343709    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k5b5s"
Nov 03 20:00:30.345345 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.345327    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vbl6p\""
Nov 03 20:00:30.345450 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.345419    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:00:30.345503 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.345489    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 20:00:30.345714 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.345692    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 20:00:30.346101 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.346073    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:00:30.346198 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:30.346169    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6crz" podUID="6e8b3941-ab89-4838-bf55-6e9521ed259c"
Nov 03 20:00:30.346939 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.346916    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 20:00:30.348211 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.347377    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 20:00:30.348211 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.347444    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lxhjg\""
Nov 03 20:00:30.349894 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.349873    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:00:30.351038 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.351021    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 20:00:30.352137 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.352093    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.354490 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.354469    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:00:30.354586 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:30.354543    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-286gh" podUID="3492be51-5e92-4369-8fc5-85d5ce0523fd"
Nov 03 20:00:30.357378 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.357357    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g2rp5"
Nov 03 20:00:30.359834 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.359549    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.363183 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.363159    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tt4df"
Nov 03 20:00:30.363649 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.363629    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/07e7cf29-701a-4c9d-a3ad-12d4846157c2-agent-certs\") pod \"konnectivity-agent-k5b5s\" (UID: \"07e7cf29-701a-4c9d-a3ad-12d4846157c2\") " pod="kube-system/konnectivity-agent-k5b5s"
Nov 03 20:00:30.363743 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.363674    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9d61c96a-2363-47b2-949a-300d9b084779-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:00:30.363857 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.363802    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-var-lib-kubelet\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.363971 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.363872    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-multus-socket-dir-parent\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.363971 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.363899    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-multus-daemon-config\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.363971 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.363921    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mlgw\" (UniqueName: \"kubernetes.io/projected/c5965c2f-be74-431c-a8ff-86598070072a-kube-api-access-6mlgw\") pod \"iptables-alerter-v9qll\" (UID: \"c5965c2f-be74-431c-a8ff-86598070072a\") " pod="openshift-network-operator/iptables-alerter-v9qll"
Nov 03 20:00:30.363971 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.363962    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/07e7cf29-701a-4c9d-a3ad-12d4846157c2-konnectivity-ca\") pod \"konnectivity-agent-k5b5s\" (UID: \"07e7cf29-701a-4c9d-a3ad-12d4846157c2\") " pod="kube-system/konnectivity-agent-k5b5s"
Nov 03 20:00:30.364184 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.363998    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpxxg\" (UniqueName: \"kubernetes.io/projected/6e8b3941-ab89-4838-bf55-6e9521ed259c-kube-api-access-dpxxg\") pod \"network-check-target-h6crz\" (UID: \"6e8b3941-ab89-4838-bf55-6e9521ed259c\") " pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:00:30.364184 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.364026    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d61c96a-2363-47b2-949a-300d9b084779-system-cni-dir\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:00:30.364184 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.364061    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-var-lib-cni-multus\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.364184 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.364098    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-multus-conf-dir\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.364184 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.364126    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d61c96a-2363-47b2-949a-300d9b084779-cni-binary-copy\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:00:30.364184 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.364149    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5q9m\" (UniqueName: \"kubernetes.io/projected/9d61c96a-2363-47b2-949a-300d9b084779-kube-api-access-j5q9m\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:00:30.364184 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.364164    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-system-cni-dir\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.364184 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.364178    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-run-netns\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.364534 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.364193    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-var-lib-cni-bin\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.364534 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.364225    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjlck\" (UniqueName: \"kubernetes.io/projected/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-kube-api-access-tjlck\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.364534 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.364242    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5965c2f-be74-431c-a8ff-86598070072a-host-slash\") pod \"iptables-alerter-v9qll\" (UID: \"c5965c2f-be74-431c-a8ff-86598070072a\") " pod="openshift-network-operator/iptables-alerter-v9qll"
Nov 03 20:00:30.364534 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.364257    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d61c96a-2363-47b2-949a-300d9b084779-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:00:30.364534 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.364271    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-hostroot\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.364534 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.364286    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-run-multus-certs\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.364534 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.364301    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-etc-kubernetes\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.364534 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.364339    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-run-k8s-cni-cncf-io\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.364534 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.364363    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d61c96a-2363-47b2-949a-300d9b084779-cnibin\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:00:30.364534 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.364388    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9d61c96a-2363-47b2-949a-300d9b084779-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:00:30.364534 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.364411    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-multus-cni-dir\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.364534 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.364429    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-cnibin\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.364534 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.364443    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-cni-binary-copy\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.364534 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.364460    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c5965c2f-be74-431c-a8ff-86598070072a-iptables-alerter-script\") pod \"iptables-alerter-v9qll\" (UID: \"c5965c2f-be74-431c-a8ff-86598070072a\") " pod="openshift-network-operator/iptables-alerter-v9qll"
Nov 03 20:00:30.367033 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.367008    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:00:30.377690 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.370330    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.377690 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.374215    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 20:00:30.378793 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.378771    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:30.378894 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:30.378838    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8wrn" podUID="0181bad8-3459-4c64-8e31-12395fd4c6c7"
Nov 03 20:00:30.401505 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.401484    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jtbvt"
Nov 03 20:00:30.412841 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.412793    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jtbvt"
Nov 03 20:00:30.452642 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.452620    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 20:00:30.454101 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.454078    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 20:00:30.454205 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.454120    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 20:00:30.454267 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.454226    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 20:00:30.454325 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.454294    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 20:00:30.454404 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.454387    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 20:00:30.454549 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.454532    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 20:00:30.454667 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.454571    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 20:00:30.454667 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.454513    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 20:00:30.454667 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.454615    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 20:00:30.454891 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.454750    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-whl79\""
Nov 03 20:00:30.454891 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.454756    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zfqpt\""
Nov 03 20:00:30.454891 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.454786    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 20:00:30.454891 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.454795    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s55dl\""
Nov 03 20:00:30.454891 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.454812    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 20:00:30.454891 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.454816    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 20:00:30.454891 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.454815    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 20:00:30.454891 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.454861    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 20:00:30.454891 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.454865    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 20:00:30.454891 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.454864    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kmt87\""
Nov 03 20:00:30.455806 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.455786    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:00:30.455806 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.455802    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8fc4f\""
Nov 03 20:00:30.455937 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.455811    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 20:00:30.455984 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.455956    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 20:00:30.455984 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.455974    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vlwgf\""
Nov 03 20:00:30.456071 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.456022    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 20:00:30.456114 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.456099    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 20:00:30.456160 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.456112    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 20:00:30.456160 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.456144    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rnml9\""
Nov 03 20:00:30.456264 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.456159    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 20:00:30.465436 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.464869    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-run-systemd\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.465436 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.464901    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-device-dir\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:00:30.465436 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.464937    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0181bad8-3459-4c64-8e31-12395fd4c6c7-original-pull-secret\") pod \"global-pull-secret-syncer-x8wrn\" (UID: \"0181bad8-3459-4c64-8e31-12395fd4c6c7\") " pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:30.465436 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465005    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-multus-conf-dir\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.465436 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465042    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d61c96a-2363-47b2-949a-300d9b084779-cni-binary-copy\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:00:30.465436 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465083    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5q9m\" (UniqueName: \"kubernetes.io/projected/9d61c96a-2363-47b2-949a-300d9b084779-kube-api-access-j5q9m\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:00:30.465436 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465227    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-system-cni-dir\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.465436 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465311    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-system-cni-dir\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.465436 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465348    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-run-netns\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.465436 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465392    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-run-netns\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.465436 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465423    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-node-log\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.465951 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465468    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-hostroot\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.465951 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465495    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-run-multus-certs\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.465951 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465514    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b8s7\" (UniqueName: \"kubernetes.io/projected/368fd902-a222-4a5b-9bbf-a8decb45ad31-kube-api-access-7b8s7\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.465951 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465530    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-systemd-units\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.465951 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465551    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/626900f3-1832-46a5-8c82-70d09284614c-ovnkube-config\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.465951 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465589    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkrkh\" (UniqueName: \"kubernetes.io/projected/626900f3-1832-46a5-8c82-70d09284614c-kube-api-access-pkrkh\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.465951 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465592    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d61c96a-2363-47b2-949a-300d9b084779-cni-binary-copy\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:00:30.465951 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465605    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:00:30.465951 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465620    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:00:30.465951 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465659    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-sysconfig\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.465951 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465697    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-hostroot\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.465951 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465697    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-run-multus-certs\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.465951 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465750    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-multus-cni-dir\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.465951 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465757    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-multus-conf-dir\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.465951 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465828    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-sys\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.465951 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465863    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-tuned\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.465951 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465890    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-cnibin\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.466663 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465928    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-kubernetes\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.466663 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465946    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-multus-cni-dir\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.466663 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465993    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-cnibin\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.466663 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.465943    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-sysctl-conf\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.466663 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.466121    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/368fd902-a222-4a5b-9bbf-a8decb45ad31-tmp\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.466663 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.466148    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv682\" (UniqueName: \"kubernetes.io/projected/78003043-36f1-452a-8ccf-37b574877715-kube-api-access-gv682\") pod \"node-ca-tt4df\" (UID: \"78003043-36f1-452a-8ccf-37b574877715\") " pod="openshift-image-registry/node-ca-tt4df"
Nov 03 20:00:30.466663 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.466202    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/07e7cf29-701a-4c9d-a3ad-12d4846157c2-agent-certs\") pod \"konnectivity-agent-k5b5s\" (UID: \"07e7cf29-701a-4c9d-a3ad-12d4846157c2\") " pod="kube-system/konnectivity-agent-k5b5s"
Nov 03 20:00:30.466663 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.466254    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9d61c96a-2363-47b2-949a-300d9b084779-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:00:30.466663 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.466285    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrkkm\" (UniqueName: \"kubernetes.io/projected/71697994-0833-4570-83d3-afa5739725e9-kube-api-access-rrkkm\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:00:30.466663 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.466305    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0181bad8-3459-4c64-8e31-12395fd4c6c7-dbus\") pod \"global-pull-secret-syncer-x8wrn\" (UID: \"0181bad8-3459-4c64-8e31-12395fd4c6c7\") " pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:30.466663 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.466351    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-etc-openvswitch\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.466663 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.466490    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-run-ovn\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.466663 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.466558    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-multus-socket-dir-parent\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.466663 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.466603    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-multus-daemon-config\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.466663 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.466655    2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 20:00:30.466663 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.466671    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-multus-socket-dir-parent\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.467511 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.466724    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-sys-fs\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:00:30.467511 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.466748    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0181bad8-3459-4c64-8e31-12395fd4c6c7-kubelet-config\") pod \"global-pull-secret-syncer-x8wrn\" (UID: \"0181bad8-3459-4c64-8e31-12395fd4c6c7\") " pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:30.467511 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.466777    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9d61c96a-2363-47b2-949a-300d9b084779-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:00:30.467511 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.466793    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-lib-modules\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.467511 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.466817    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/78003043-36f1-452a-8ccf-37b574877715-serviceca\") pod \"node-ca-tt4df\" (UID: \"78003043-36f1-452a-8ccf-37b574877715\") " pod="openshift-image-registry/node-ca-tt4df"
Nov 03 20:00:30.467511 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.466847    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mlgw\" (UniqueName: \"kubernetes.io/projected/c5965c2f-be74-431c-a8ff-86598070072a-kube-api-access-6mlgw\") pod \"iptables-alerter-v9qll\" (UID: \"c5965c2f-be74-431c-a8ff-86598070072a\") " pod="openshift-network-operator/iptables-alerter-v9qll"
Nov 03 20:00:30.467511 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.466879    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d61c96a-2363-47b2-949a-300d9b084779-system-cni-dir\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:00:30.467511 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.466932    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-host\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.467511 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.466956    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-run-netns\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.467511 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467006    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-run-openvswitch\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.467511 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467032    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d61c96a-2363-47b2-949a-300d9b084779-system-cni-dir\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:00:30.467511 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467079    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-run-ovn-kubernetes\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.467511 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467104    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/153d1ce3-22c2-444c-9bd2-ee91ff044d29-hosts-file\") pod \"node-resolver-g2rp5\" (UID: \"153d1ce3-22c2-444c-9bd2-ee91ff044d29\") " pod="openshift-dns/node-resolver-g2rp5"
Nov 03 20:00:30.467511 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467136    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g525b\" (UniqueName: \"kubernetes.io/projected/153d1ce3-22c2-444c-9bd2-ee91ff044d29-kube-api-access-g525b\") pod \"node-resolver-g2rp5\" (UID: \"153d1ce3-22c2-444c-9bd2-ee91ff044d29\") " pod="openshift-dns/node-resolver-g2rp5"
Nov 03 20:00:30.467511 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467172    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-var-lib-cni-multus\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.467511 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467191    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-var-lib-cni-bin\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.467511 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467214    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjlck\" (UniqueName: \"kubernetes.io/projected/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-kube-api-access-tjlck\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.468377 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467230    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5965c2f-be74-431c-a8ff-86598070072a-host-slash\") pod \"iptables-alerter-v9qll\" (UID: \"c5965c2f-be74-431c-a8ff-86598070072a\") " pod="openshift-network-operator/iptables-alerter-v9qll"
Nov 03 20:00:30.468377 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467247    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d61c96a-2363-47b2-949a-300d9b084779-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:00:30.468377 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467282    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-etc-kubernetes\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.468377 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467301    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-registration-dir\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:00:30.468377 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467319    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.468377 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467352    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/626900f3-1832-46a5-8c82-70d09284614c-ovn-node-metrics-cert\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.468377 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467370    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-run-k8s-cni-cncf-io\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.468377 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467398    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/153d1ce3-22c2-444c-9bd2-ee91ff044d29-tmp-dir\") pod \"node-resolver-g2rp5\" (UID: \"153d1ce3-22c2-444c-9bd2-ee91ff044d29\") " pod="openshift-dns/node-resolver-g2rp5"
Nov 03 20:00:30.468377 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467438    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3492be51-5e92-4369-8fc5-85d5ce0523fd-metrics-certs\") pod \"network-metrics-daemon-286gh\" (UID: \"3492be51-5e92-4369-8fc5-85d5ce0523fd\") " pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:00:30.468377 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467454    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-systemd\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.468377 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467470    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-var-lib-kubelet\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.468377 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467485    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-kubelet\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.468377 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467500    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-log-socket\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.468377 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467507    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-multus-daemon-config\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.468377 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467516    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d61c96a-2363-47b2-949a-300d9b084779-cnibin\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:00:30.468377 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467539    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d61c96a-2363-47b2-949a-300d9b084779-cnibin\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:00:30.468377 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467564    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5965c2f-be74-431c-a8ff-86598070072a-host-slash\") pod \"iptables-alerter-v9qll\" (UID: \"c5965c2f-be74-431c-a8ff-86598070072a\") " pod="openshift-network-operator/iptables-alerter-v9qll"
Nov 03 20:00:30.468980 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467567    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9d61c96a-2363-47b2-949a-300d9b084779-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:00:30.468980 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467585    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-etc-kubernetes\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.468980 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467604    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d61c96a-2363-47b2-949a-300d9b084779-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:00:30.468980 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467614    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k9l5\" (UniqueName: \"kubernetes.io/projected/3492be51-5e92-4369-8fc5-85d5ce0523fd-kube-api-access-5k9l5\") pod \"network-metrics-daemon-286gh\" (UID: \"3492be51-5e92-4369-8fc5-85d5ce0523fd\") " pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:00:30.468980 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467631    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-var-lib-openvswitch\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.468980 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467648    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-cni-bin\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.468980 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467739    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-cni-netd\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.468980 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467757    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-run-k8s-cni-cncf-io\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.468980 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467795    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-cni-binary-copy\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.468980 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467834    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-var-lib-cni-multus\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.468980 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467825    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78003043-36f1-452a-8ccf-37b574877715-host\") pod \"node-ca-tt4df\" (UID: \"78003043-36f1-452a-8ccf-37b574877715\") " pod="openshift-image-registry/node-ca-tt4df"
Nov 03 20:00:30.468980 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467891    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-var-lib-cni-bin\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.468980 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467928    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c5965c2f-be74-431c-a8ff-86598070072a-iptables-alerter-script\") pod \"iptables-alerter-v9qll\" (UID: \"c5965c2f-be74-431c-a8ff-86598070072a\") " pod="openshift-network-operator/iptables-alerter-v9qll"
Nov 03 20:00:30.468980 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467963    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-var-lib-kubelet\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.468980 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.467993    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-sysctl-d\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.468980 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.468020    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-modprobe-d\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.468980 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.468076    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-slash\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.469594 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.468096    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9d61c96a-2363-47b2-949a-300d9b084779-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:00:30.469594 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.468120    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/626900f3-1832-46a5-8c82-70d09284614c-env-overrides\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.469594 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.468154    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/626900f3-1832-46a5-8c82-70d09284614c-ovnkube-script-lib\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.469594 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.468177    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-var-lib-kubelet\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.469594 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.468307    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-cni-binary-copy\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.469594 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.468324    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/07e7cf29-701a-4c9d-a3ad-12d4846157c2-konnectivity-ca\") pod \"konnectivity-agent-k5b5s\" (UID: \"07e7cf29-701a-4c9d-a3ad-12d4846157c2\") " pod="kube-system/konnectivity-agent-k5b5s"
Nov 03 20:00:30.469594 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.468373    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxxg\" (UniqueName: \"kubernetes.io/projected/6e8b3941-ab89-4838-bf55-6e9521ed259c-kube-api-access-dpxxg\") pod \"network-check-target-h6crz\" (UID: \"6e8b3941-ab89-4838-bf55-6e9521ed259c\") " pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:00:30.469594 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.468504    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-socket-dir\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:00:30.469594 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.468535    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-run\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.469594 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.468746    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c5965c2f-be74-431c-a8ff-86598070072a-iptables-alerter-script\") pod \"iptables-alerter-v9qll\" (UID: \"c5965c2f-be74-431c-a8ff-86598070072a\") " pod="openshift-network-operator/iptables-alerter-v9qll"
Nov 03 20:00:30.469594 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.468926    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/07e7cf29-701a-4c9d-a3ad-12d4846157c2-konnectivity-ca\") pod \"konnectivity-agent-k5b5s\" (UID: \"07e7cf29-701a-4c9d-a3ad-12d4846157c2\") " pod="kube-system/konnectivity-agent-k5b5s"
Nov 03 20:00:30.471157 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.471140    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/07e7cf29-701a-4c9d-a3ad-12d4846157c2-agent-certs\") pod \"konnectivity-agent-k5b5s\" (UID: \"07e7cf29-701a-4c9d-a3ad-12d4846157c2\") " pod="kube-system/konnectivity-agent-k5b5s"
Nov 03 20:00:30.478187 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.478162    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5q9m\" (UniqueName: \"kubernetes.io/projected/9d61c96a-2363-47b2-949a-300d9b084779-kube-api-access-j5q9m\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:00:30.478618 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:30.478592    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:00:30.478618 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:30.478619    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:00:30.478755 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:30.478632    2575 projected.go:196] Error preparing data for projected volume kube-api-access-dpxxg for pod openshift-network-diagnostics/network-check-target-h6crz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:00:30.478755 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:30.478727    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e8b3941-ab89-4838-bf55-6e9521ed259c-kube-api-access-dpxxg podName:6e8b3941-ab89-4838-bf55-6e9521ed259c nodeName:}" failed. No retries permitted until 2026-11-03 20:00:30.978701228 +0000 UTC m=+2.971857622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dpxxg" (UniqueName: "kubernetes.io/projected/6e8b3941-ab89-4838-bf55-6e9521ed259c-kube-api-access-dpxxg") pod "network-check-target-h6crz" (UID: "6e8b3941-ab89-4838-bf55-6e9521ed259c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:00:30.479711 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.479664    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjlck\" (UniqueName: \"kubernetes.io/projected/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-kube-api-access-tjlck\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.480091 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.480073    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mlgw\" (UniqueName: \"kubernetes.io/projected/c5965c2f-be74-431c-a8ff-86598070072a-kube-api-access-6mlgw\") pod \"iptables-alerter-v9qll\" (UID: \"c5965c2f-be74-431c-a8ff-86598070072a\") " pod="openshift-network-operator/iptables-alerter-v9qll"
Nov 03 20:00:30.525039 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.524992    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal" event={"ID":"b45f8e0dedc76d41e693c990769490b8","Type":"ContainerStarted","Data":"f47a37dac525b92bc09e460b56f9cd458f1074daf1136c74e3f6247f5b025d21"}
Nov 03 20:00:30.526159 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.526121    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal" event={"ID":"cc3d49b24d21c77c008662d39f9bb541","Type":"ContainerStarted","Data":"072b4d3337117c5ea0ce179d4b73bad04b5790a500582f1468d6595e2fed9a06"}
Nov 03 20:00:30.569285 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569257    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-modprobe-d\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.569433 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569290    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-slash\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.569433 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569314    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/626900f3-1832-46a5-8c82-70d09284614c-env-overrides\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.569433 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569338    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/626900f3-1832-46a5-8c82-70d09284614c-ovnkube-script-lib\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.569433 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569374    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-socket-dir\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:00:30.569433 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569394    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-run\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.569433 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569416    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-run-systemd\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.569688 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569434    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-modprobe-d\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.569688 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569439    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-device-dir\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:00:30.569688 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569475    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0181bad8-3459-4c64-8e31-12395fd4c6c7-original-pull-secret\") pod \"global-pull-secret-syncer-x8wrn\" (UID: \"0181bad8-3459-4c64-8e31-12395fd4c6c7\") " pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:30.569688 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569482    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-device-dir\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:00:30.569688 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569507    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-node-log\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.569688 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569535    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-slash\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.569688 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569535    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b8s7\" (UniqueName: \"kubernetes.io/projected/368fd902-a222-4a5b-9bbf-a8decb45ad31-kube-api-access-7b8s7\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.569688 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569570    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-systemd-units\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.569688 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569593    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/626900f3-1832-46a5-8c82-70d09284614c-ovnkube-config\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.569688 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569616    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkrkh\" (UniqueName: \"kubernetes.io/projected/626900f3-1832-46a5-8c82-70d09284614c-kube-api-access-pkrkh\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.569688 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569657    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:00:30.569688 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569681    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:00:30.570097 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569706    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-sysconfig\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.570097 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569731    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-sys\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.570097 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569753    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-tuned\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.570097 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569777    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-kubernetes\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.570097 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569804    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-sysctl-conf\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.570097 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569825    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/368fd902-a222-4a5b-9bbf-a8decb45ad31-tmp\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.570097 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:30.569831    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:00:30.570097 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569850    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv682\" (UniqueName: \"kubernetes.io/projected/78003043-36f1-452a-8ccf-37b574877715-kube-api-access-gv682\") pod \"node-ca-tt4df\" (UID: \"78003043-36f1-452a-8ccf-37b574877715\") " pod="openshift-image-registry/node-ca-tt4df"
Nov 03 20:00:30.570097 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569876    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrkkm\" (UniqueName: \"kubernetes.io/projected/71697994-0833-4570-83d3-afa5739725e9-kube-api-access-rrkkm\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:00:30.570097 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:30.569894    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0181bad8-3459-4c64-8e31-12395fd4c6c7-original-pull-secret podName:0181bad8-3459-4c64-8e31-12395fd4c6c7 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:31.069875413 +0000 UTC m=+3.063031807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0181bad8-3459-4c64-8e31-12395fd4c6c7-original-pull-secret") pod "global-pull-secret-syncer-x8wrn" (UID: "0181bad8-3459-4c64-8e31-12395fd4c6c7") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:00:30.570097 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569920    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0181bad8-3459-4c64-8e31-12395fd4c6c7-dbus\") pod \"global-pull-secret-syncer-x8wrn\" (UID: \"0181bad8-3459-4c64-8e31-12395fd4c6c7\") " pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:30.570097 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569946    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-etc-openvswitch\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.570097 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569971    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-run-ovn\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.570097 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.569999    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-sys-fs\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:00:30.570097 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570024    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0181bad8-3459-4c64-8e31-12395fd4c6c7-kubelet-config\") pod \"global-pull-secret-syncer-x8wrn\" (UID: \"0181bad8-3459-4c64-8e31-12395fd4c6c7\") " pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:30.570097 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570050    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-lib-modules\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.570097 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570074    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/78003043-36f1-452a-8ccf-37b574877715-serviceca\") pod \"node-ca-tt4df\" (UID: \"78003043-36f1-452a-8ccf-37b574877715\") " pod="openshift-image-registry/node-ca-tt4df"
Nov 03 20:00:30.570893 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570099    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-host\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.570893 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570125    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-run-netns\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.570893 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570146    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-run-openvswitch\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.570893 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570168    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-run-ovn-kubernetes\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.570893 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570193    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/153d1ce3-22c2-444c-9bd2-ee91ff044d29-hosts-file\") pod \"node-resolver-g2rp5\" (UID: \"153d1ce3-22c2-444c-9bd2-ee91ff044d29\") " pod="openshift-dns/node-resolver-g2rp5"
Nov 03 20:00:30.570893 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570220    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g525b\" (UniqueName: \"kubernetes.io/projected/153d1ce3-22c2-444c-9bd2-ee91ff044d29-kube-api-access-g525b\") pod \"node-resolver-g2rp5\" (UID: \"153d1ce3-22c2-444c-9bd2-ee91ff044d29\") " pod="openshift-dns/node-resolver-g2rp5"
Nov 03 20:00:30.570893 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570256    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-registration-dir\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:00:30.570893 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570284    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.570893 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570295    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:00:30.570893 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570312    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/626900f3-1832-46a5-8c82-70d09284614c-ovn-node-metrics-cert\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.570893 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570360    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/153d1ce3-22c2-444c-9bd2-ee91ff044d29-tmp-dir\") pod \"node-resolver-g2rp5\" (UID: \"153d1ce3-22c2-444c-9bd2-ee91ff044d29\") " pod="openshift-dns/node-resolver-g2rp5"
Nov 03 20:00:30.570893 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570392    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3492be51-5e92-4369-8fc5-85d5ce0523fd-metrics-certs\") pod \"network-metrics-daemon-286gh\" (UID: \"3492be51-5e92-4369-8fc5-85d5ce0523fd\") " pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:00:30.570893 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570421    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-systemd\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.570893 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570449    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-var-lib-kubelet\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.570893 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570486    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-kubelet\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.570893 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570510    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/626900f3-1832-46a5-8c82-70d09284614c-env-overrides\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.570893 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570514    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-log-socket\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.571742 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570547    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k9l5\" (UniqueName: \"kubernetes.io/projected/3492be51-5e92-4369-8fc5-85d5ce0523fd-kube-api-access-5k9l5\") pod \"network-metrics-daemon-286gh\" (UID: \"3492be51-5e92-4369-8fc5-85d5ce0523fd\") " pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:00:30.571742 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570576    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-var-lib-openvswitch\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.571742 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570602    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-cni-bin\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.571742 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570628    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:00:30.571742 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570651    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-cni-netd\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.571742 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570682    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78003043-36f1-452a-8ccf-37b574877715-host\") pod \"node-ca-tt4df\" (UID: \"78003043-36f1-452a-8ccf-37b574877715\") " pod="openshift-image-registry/node-ca-tt4df"
Nov 03 20:00:30.571742 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570695    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-sysconfig\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.571742 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570712    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-sysctl-d\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.571742 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570762    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-sys\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.571742 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570837    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-sysctl-conf\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.571742 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:30.570847    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:00:30.571742 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570879    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-sysctl-d\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.571742 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:30.570904    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3492be51-5e92-4369-8fc5-85d5ce0523fd-metrics-certs podName:3492be51-5e92-4369-8fc5-85d5ce0523fd nodeName:}" failed. No retries permitted until 2026-11-03 20:00:31.070887837 +0000 UTC m=+3.064044231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3492be51-5e92-4369-8fc5-85d5ce0523fd-metrics-certs") pod "network-metrics-daemon-286gh" (UID: "3492be51-5e92-4369-8fc5-85d5ce0523fd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:00:30.571742 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570953    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-var-lib-openvswitch\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.571742 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.570992    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-var-lib-kubelet\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.571742 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.571001    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-cni-bin\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.571742 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.571038    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-cni-netd\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.572501 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.571060    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-systemd\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.572501 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.571097    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78003043-36f1-452a-8ccf-37b574877715-host\") pod \"node-ca-tt4df\" (UID: \"78003043-36f1-452a-8ccf-37b574877715\") " pod="openshift-image-registry/node-ca-tt4df"
Nov 03 20:00:30.572501 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.571110    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-kubelet\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.572501 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.571165    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-log-socket\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.572501 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.571163    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-run-systemd\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.572501 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.571221    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-kubernetes\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.572501 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.571236    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-systemd-units\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.572501 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.571269    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/153d1ce3-22c2-444c-9bd2-ee91ff044d29-tmp-dir\") pod \"node-resolver-g2rp5\" (UID: \"153d1ce3-22c2-444c-9bd2-ee91ff044d29\") " pod="openshift-dns/node-resolver-g2rp5"
Nov 03 20:00:30.572501 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.571421    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-socket-dir\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:00:30.572501 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.571622    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-etc-openvswitch\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.572501 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.571663    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/626900f3-1832-46a5-8c82-70d09284614c-ovnkube-script-lib\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.572501 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.571696    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-node-log\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.572501 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.571757    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/626900f3-1832-46a5-8c82-70d09284614c-ovnkube-config\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.572501 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.571764    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0181bad8-3459-4c64-8e31-12395fd4c6c7-dbus\") pod \"global-pull-secret-syncer-x8wrn\" (UID: \"0181bad8-3459-4c64-8e31-12395fd4c6c7\") " pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:30.572501 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.571765    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-run\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.572501 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.571886    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/78003043-36f1-452a-8ccf-37b574877715-serviceca\") pod \"node-ca-tt4df\" (UID: \"78003043-36f1-452a-8ccf-37b574877715\") " pod="openshift-image-registry/node-ca-tt4df"
Nov 03 20:00:30.572501 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.571938    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-run-ovn-kubernetes\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.572501 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.571957    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-run-ovn\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.573337 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.572044    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-run-netns\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.573337 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.572086    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-sys-fs\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:00:30.573337 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.572099    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-host\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.573337 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.572109    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-lib-modules\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.573337 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.572118    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-registration-dir\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:00:30.573337 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.572128    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-run-openvswitch\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.573337 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.572152    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0181bad8-3459-4c64-8e31-12395fd4c6c7-kubelet-config\") pod \"global-pull-secret-syncer-x8wrn\" (UID: \"0181bad8-3459-4c64-8e31-12395fd4c6c7\") " pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:30.573337 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.572153    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.573337 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.572184    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/153d1ce3-22c2-444c-9bd2-ee91ff044d29-hosts-file\") pod \"node-resolver-g2rp5\" (UID: \"153d1ce3-22c2-444c-9bd2-ee91ff044d29\") " pod="openshift-dns/node-resolver-g2rp5"
Nov 03 20:00:30.573337 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.573186    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/626900f3-1832-46a5-8c82-70d09284614c-ovn-node-metrics-cert\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.574310 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.574237    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/368fd902-a222-4a5b-9bbf-a8decb45ad31-tmp\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.574632 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.574612    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-tuned\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.585558 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.585434    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b8s7\" (UniqueName: \"kubernetes.io/projected/368fd902-a222-4a5b-9bbf-a8decb45ad31-kube-api-access-7b8s7\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.587651 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.587624    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrkkm\" (UniqueName: \"kubernetes.io/projected/71697994-0833-4570-83d3-afa5739725e9-kube-api-access-rrkkm\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:00:30.588101 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.588077    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkrkh\" (UniqueName: \"kubernetes.io/projected/626900f3-1832-46a5-8c82-70d09284614c-kube-api-access-pkrkh\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.588238 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.588195    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g525b\" (UniqueName: \"kubernetes.io/projected/153d1ce3-22c2-444c-9bd2-ee91ff044d29-kube-api-access-g525b\") pod \"node-resolver-g2rp5\" (UID: \"153d1ce3-22c2-444c-9bd2-ee91ff044d29\") " pod="openshift-dns/node-resolver-g2rp5"
Nov 03 20:00:30.588861 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.588527    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k9l5\" (UniqueName: \"kubernetes.io/projected/3492be51-5e92-4369-8fc5-85d5ce0523fd-kube-api-access-5k9l5\") pod \"network-metrics-daemon-286gh\" (UID: \"3492be51-5e92-4369-8fc5-85d5ce0523fd\") " pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:00:30.588861 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.588820    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv682\" (UniqueName: \"kubernetes.io/projected/78003043-36f1-452a-8ccf-37b574877715-kube-api-access-gv682\") pod \"node-ca-tt4df\" (UID: \"78003043-36f1-452a-8ccf-37b574877715\") " pod="openshift-image-registry/node-ca-tt4df"
Nov 03 20:00:30.658504 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.658466    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v9qll"
Nov 03 20:00:30.668154 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.668132    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k5b5s"
Nov 03 20:00:30.668475 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:30.668453    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5965c2f_be74_431c_a8ff_86598070072a.slice/crio-1cc9c89c3e02a465870ec2e297a5829ce2d3201cd7547f8409e67fdf0eb3a34f WatchSource:0}: Error finding container 1cc9c89c3e02a465870ec2e297a5829ce2d3201cd7547f8409e67fdf0eb3a34f: Status 404 returned error can't find the container with id 1cc9c89c3e02a465870ec2e297a5829ce2d3201cd7547f8409e67fdf0eb3a34f
Nov 03 20:00:30.677962 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:30.677935    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e7cf29_701a_4c9d_a3ad_12d4846157c2.slice/crio-f04e18246bd8e09ce84e96e49436083094cdbe6c95ec881d4748a5645dd718dc WatchSource:0}: Error finding container f04e18246bd8e09ce84e96e49436083094cdbe6c95ec881d4748a5645dd718dc: Status 404 returned error can't find the container with id f04e18246bd8e09ce84e96e49436083094cdbe6c95ec881d4748a5645dd718dc
Nov 03 20:00:30.697064 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.697044    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mq5xz"
Nov 03 20:00:30.700278 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.700258    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:00:30.703121 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.703100    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g2rp5"
Nov 03 20:00:30.707189 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.707169    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:00:30.708573 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:30.708547    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4c4fc1_a4c5_4f00_97bc_b201eede1226.slice/crio-c6bb9956239dd70aaa24e2daa761f3809e82a5c3d2ac772b236ae77199e8d584 WatchSource:0}: Error finding container c6bb9956239dd70aaa24e2daa761f3809e82a5c3d2ac772b236ae77199e8d584: Status 404 returned error can't find the container with id c6bb9956239dd70aaa24e2daa761f3809e82a5c3d2ac772b236ae77199e8d584
Nov 03 20:00:30.715581 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:30.715557    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d61c96a_2363_47b2_949a_300d9b084779.slice/crio-4812e27e425a4a2b5c830a46560c0f9fee370441a6f57cd7a1c1eb5ab2ab54fa WatchSource:0}: Error finding container 4812e27e425a4a2b5c830a46560c0f9fee370441a6f57cd7a1c1eb5ab2ab54fa: Status 404 returned error can't find the container with id 4812e27e425a4a2b5c830a46560c0f9fee370441a6f57cd7a1c1eb5ab2ab54fa
Nov 03 20:00:30.716567 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.716542    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tt4df"
Nov 03 20:00:30.716984 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:30.716964    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153d1ce3_22c2_444c_9bd2_ee91ff044d29.slice/crio-66ee5f1c87b8eb08c4370e0e3faef18cef3bf7921d8a748e5845e197c1337fb7 WatchSource:0}: Error finding container 66ee5f1c87b8eb08c4370e0e3faef18cef3bf7921d8a748e5845e197c1337fb7: Status 404 returned error can't find the container with id 66ee5f1c87b8eb08c4370e0e3faef18cef3bf7921d8a748e5845e197c1337fb7
Nov 03 20:00:30.719128 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:30.719085    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod368fd902_a222_4a5b_9bbf_a8decb45ad31.slice/crio-ad4f11a7f586fa65711327857b5a98433e079cfc91980384035276ae89547f21 WatchSource:0}: Error finding container ad4f11a7f586fa65711327857b5a98433e079cfc91980384035276ae89547f21: Status 404 returned error can't find the container with id ad4f11a7f586fa65711327857b5a98433e079cfc91980384035276ae89547f21
Nov 03 20:00:30.727445 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.727422    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:00:30.729325 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:30.729295    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78003043_36f1_452a_8ccf_37b574877715.slice/crio-1bd82a08adbfcc7c587ec320745a85f7621d600dc1ceb21ab2e4329239ab1781 WatchSource:0}: Error finding container 1bd82a08adbfcc7c587ec320745a85f7621d600dc1ceb21ab2e4329239ab1781: Status 404 returned error can't find the container with id 1bd82a08adbfcc7c587ec320745a85f7621d600dc1ceb21ab2e4329239ab1781
Nov 03 20:00:30.738000 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:30.737981    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71697994_0833_4570_83d3_afa5739725e9.slice/crio-a24ce8f1bb29443403798955cc34c0a42dee8258d23bda54a0757d54f49edcb1 WatchSource:0}: Error finding container a24ce8f1bb29443403798955cc34c0a42dee8258d23bda54a0757d54f49edcb1: Status 404 returned error can't find the container with id a24ce8f1bb29443403798955cc34c0a42dee8258d23bda54a0757d54f49edcb1
Nov 03 20:00:30.745720 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:30.745701    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:30.755518 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:00:30.755497    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod626900f3_1832_46a5_8c82_70d09284614c.slice/crio-6d5b9e407b152f10b72d3b5b458318d2f902da2c03f303f1476c8f0f66f07578 WatchSource:0}: Error finding container 6d5b9e407b152f10b72d3b5b458318d2f902da2c03f303f1476c8f0f66f07578: Status 404 returned error can't find the container with id 6d5b9e407b152f10b72d3b5b458318d2f902da2c03f303f1476c8f0f66f07578
Nov 03 20:00:31.073912 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:31.073821    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxxg\" (UniqueName: \"kubernetes.io/projected/6e8b3941-ab89-4838-bf55-6e9521ed259c-kube-api-access-dpxxg\") pod \"network-check-target-h6crz\" (UID: \"6e8b3941-ab89-4838-bf55-6e9521ed259c\") " pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:00:31.073912 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:31.073882    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0181bad8-3459-4c64-8e31-12395fd4c6c7-original-pull-secret\") pod \"global-pull-secret-syncer-x8wrn\" (UID: \"0181bad8-3459-4c64-8e31-12395fd4c6c7\") " pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:31.074113 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:31.073948    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3492be51-5e92-4369-8fc5-85d5ce0523fd-metrics-certs\") pod \"network-metrics-daemon-286gh\" (UID: \"3492be51-5e92-4369-8fc5-85d5ce0523fd\") " pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:00:31.074113 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:31.074071    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:00:31.074225 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:31.074137    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3492be51-5e92-4369-8fc5-85d5ce0523fd-metrics-certs podName:3492be51-5e92-4369-8fc5-85d5ce0523fd nodeName:}" failed. No retries permitted until 2026-11-03 20:00:32.074118171 +0000 UTC m=+4.067274560 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3492be51-5e92-4369-8fc5-85d5ce0523fd-metrics-certs") pod "network-metrics-daemon-286gh" (UID: "3492be51-5e92-4369-8fc5-85d5ce0523fd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:00:31.074550 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:31.074527    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:00:31.074550 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:31.074550    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:00:31.074749 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:31.074562    2575 projected.go:196] Error preparing data for projected volume kube-api-access-dpxxg for pod openshift-network-diagnostics/network-check-target-h6crz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:00:31.074749 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:31.074609    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e8b3941-ab89-4838-bf55-6e9521ed259c-kube-api-access-dpxxg podName:6e8b3941-ab89-4838-bf55-6e9521ed259c nodeName:}" failed. No retries permitted until 2026-11-03 20:00:32.074596014 +0000 UTC m=+4.067752404 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dpxxg" (UniqueName: "kubernetes.io/projected/6e8b3941-ab89-4838-bf55-6e9521ed259c-kube-api-access-dpxxg") pod "network-check-target-h6crz" (UID: "6e8b3941-ab89-4838-bf55-6e9521ed259c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:00:31.074749 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:31.074694    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:00:31.074749 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:31.074733    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0181bad8-3459-4c64-8e31-12395fd4c6c7-original-pull-secret podName:0181bad8-3459-4c64-8e31-12395fd4c6c7 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:32.074720577 +0000 UTC m=+4.067876968 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0181bad8-3459-4c64-8e31-12395fd4c6c7-original-pull-secret") pod "global-pull-secret-syncer-x8wrn" (UID: "0181bad8-3459-4c64-8e31-12395fd4c6c7") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:00:31.415888 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:31.415823    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:55:31 +0000 UTC" deadline="2028-06-23 13:17:57.979265442 +0000 UTC"
Nov 03 20:00:31.415888 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:31.415888    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14345h17m26.56338203s"
Nov 03 20:00:31.517444 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:31.517409    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:31.517623 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:31.517597    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8wrn" podUID="0181bad8-3459-4c64-8e31-12395fd4c6c7"
Nov 03 20:00:31.518064 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:31.518045    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:00:31.518161 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:31.518142    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6crz" podUID="6e8b3941-ab89-4838-bf55-6e9521ed259c"
Nov 03 20:00:31.530283 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:31.530252    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v9qll" event={"ID":"c5965c2f-be74-431c-a8ff-86598070072a","Type":"ContainerStarted","Data":"1cc9c89c3e02a465870ec2e297a5829ce2d3201cd7547f8409e67fdf0eb3a34f"}
Nov 03 20:00:31.536594 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:31.535940    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal" event={"ID":"cc3d49b24d21c77c008662d39f9bb541","Type":"ContainerStarted","Data":"822a9cc5a83b8416ab77a8cc625fbb7769bd3eff45c26476a364f428d933fba1"}
Nov 03 20:00:31.538984 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:31.538918    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w72hd" event={"ID":"368fd902-a222-4a5b-9bbf-a8decb45ad31","Type":"ContainerStarted","Data":"ad4f11a7f586fa65711327857b5a98433e079cfc91980384035276ae89547f21"}
Nov 03 20:00:31.544203 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:31.544169    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mq5xz" event={"ID":"ac4c4fc1-a4c5-4f00-97bc-b201eede1226","Type":"ContainerStarted","Data":"c6bb9956239dd70aaa24e2daa761f3809e82a5c3d2ac772b236ae77199e8d584"}
Nov 03 20:00:31.546627 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:31.546600    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r" event={"ID":"626900f3-1832-46a5-8c82-70d09284614c","Type":"ContainerStarted","Data":"6d5b9e407b152f10b72d3b5b458318d2f902da2c03f303f1476c8f0f66f07578"}
Nov 03 20:00:31.549212 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:31.549188    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb" event={"ID":"71697994-0833-4570-83d3-afa5739725e9","Type":"ContainerStarted","Data":"a24ce8f1bb29443403798955cc34c0a42dee8258d23bda54a0757d54f49edcb1"}
Nov 03 20:00:31.553771 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:31.553507    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tt4df" event={"ID":"78003043-36f1-452a-8ccf-37b574877715","Type":"ContainerStarted","Data":"1bd82a08adbfcc7c587ec320745a85f7621d600dc1ceb21ab2e4329239ab1781"}
Nov 03 20:00:31.556160 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:31.556135    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g2rp5" event={"ID":"153d1ce3-22c2-444c-9bd2-ee91ff044d29","Type":"ContainerStarted","Data":"66ee5f1c87b8eb08c4370e0e3faef18cef3bf7921d8a748e5845e197c1337fb7"}
Nov 03 20:00:31.559270 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:31.559244    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86vz2" event={"ID":"9d61c96a-2363-47b2-949a-300d9b084779","Type":"ContainerStarted","Data":"4812e27e425a4a2b5c830a46560c0f9fee370441a6f57cd7a1c1eb5ab2ab54fa"}
Nov 03 20:00:31.561664 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:31.561430    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k5b5s" event={"ID":"07e7cf29-701a-4c9d-a3ad-12d4846157c2","Type":"ContainerStarted","Data":"f04e18246bd8e09ce84e96e49436083094cdbe6c95ec881d4748a5645dd718dc"}
Nov 03 20:00:32.084161 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:32.084109    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3492be51-5e92-4369-8fc5-85d5ce0523fd-metrics-certs\") pod \"network-metrics-daemon-286gh\" (UID: \"3492be51-5e92-4369-8fc5-85d5ce0523fd\") " pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:00:32.084324 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:32.084186    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxxg\" (UniqueName: \"kubernetes.io/projected/6e8b3941-ab89-4838-bf55-6e9521ed259c-kube-api-access-dpxxg\") pod \"network-check-target-h6crz\" (UID: \"6e8b3941-ab89-4838-bf55-6e9521ed259c\") " pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:00:32.084324 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:32.084249    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0181bad8-3459-4c64-8e31-12395fd4c6c7-original-pull-secret\") pod \"global-pull-secret-syncer-x8wrn\" (UID: \"0181bad8-3459-4c64-8e31-12395fd4c6c7\") " pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:32.084463 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:32.084445    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:00:32.084528 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:32.084516    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0181bad8-3459-4c64-8e31-12395fd4c6c7-original-pull-secret podName:0181bad8-3459-4c64-8e31-12395fd4c6c7 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:34.084497112 +0000 UTC m=+6.077653492 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0181bad8-3459-4c64-8e31-12395fd4c6c7-original-pull-secret") pod "global-pull-secret-syncer-x8wrn" (UID: "0181bad8-3459-4c64-8e31-12395fd4c6c7") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:00:32.085104 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:32.085084    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:00:32.085217 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:32.085143    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3492be51-5e92-4369-8fc5-85d5ce0523fd-metrics-certs podName:3492be51-5e92-4369-8fc5-85d5ce0523fd nodeName:}" failed. No retries permitted until 2026-11-03 20:00:34.085126648 +0000 UTC m=+6.078283028 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3492be51-5e92-4369-8fc5-85d5ce0523fd-metrics-certs") pod "network-metrics-daemon-286gh" (UID: "3492be51-5e92-4369-8fc5-85d5ce0523fd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:00:32.085288 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:32.085227    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:00:32.085288 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:32.085241    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:00:32.085288 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:32.085254    2575 projected.go:196] Error preparing data for projected volume kube-api-access-dpxxg for pod openshift-network-diagnostics/network-check-target-h6crz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:00:32.085458 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:32.085292    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e8b3941-ab89-4838-bf55-6e9521ed259c-kube-api-access-dpxxg podName:6e8b3941-ab89-4838-bf55-6e9521ed259c nodeName:}" failed. No retries permitted until 2026-11-03 20:00:34.085281122 +0000 UTC m=+6.078437504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dpxxg" (UniqueName: "kubernetes.io/projected/6e8b3941-ab89-4838-bf55-6e9521ed259c-kube-api-access-dpxxg") pod "network-check-target-h6crz" (UID: "6e8b3941-ab89-4838-bf55-6e9521ed259c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:00:32.416916 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:32.416800    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:55:31 +0000 UTC" deadline="2028-07-08 08:13:10.464423202 +0000 UTC"
Nov 03 20:00:32.416916 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:32.416866    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14700h12m38.047561769s"
Nov 03 20:00:32.523209 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:32.523175    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:00:32.523390 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:32.523341    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-286gh" podUID="3492be51-5e92-4369-8fc5-85d5ce0523fd"
Nov 03 20:00:33.517470 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:33.517428    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:33.517989 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:33.517556    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8wrn" podUID="0181bad8-3459-4c64-8e31-12395fd4c6c7"
Nov 03 20:00:33.517989 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:33.517980    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:00:33.518116 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:33.518093    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6crz" podUID="6e8b3941-ab89-4838-bf55-6e9521ed259c"
Nov 03 20:00:34.102598 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:34.102310    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3492be51-5e92-4369-8fc5-85d5ce0523fd-metrics-certs\") pod \"network-metrics-daemon-286gh\" (UID: \"3492be51-5e92-4369-8fc5-85d5ce0523fd\") " pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:00:34.102811 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:34.102636    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxxg\" (UniqueName: \"kubernetes.io/projected/6e8b3941-ab89-4838-bf55-6e9521ed259c-kube-api-access-dpxxg\") pod \"network-check-target-h6crz\" (UID: \"6e8b3941-ab89-4838-bf55-6e9521ed259c\") " pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:00:34.102811 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:34.102691    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0181bad8-3459-4c64-8e31-12395fd4c6c7-original-pull-secret\") pod \"global-pull-secret-syncer-x8wrn\" (UID: \"0181bad8-3459-4c64-8e31-12395fd4c6c7\") " pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:34.102951 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:34.102841    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:00:34.102951 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:34.102911    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0181bad8-3459-4c64-8e31-12395fd4c6c7-original-pull-secret podName:0181bad8-3459-4c64-8e31-12395fd4c6c7 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:38.102890945 +0000 UTC m=+10.096047336 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0181bad8-3459-4c64-8e31-12395fd4c6c7-original-pull-secret") pod "global-pull-secret-syncer-x8wrn" (UID: "0181bad8-3459-4c64-8e31-12395fd4c6c7") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:00:34.103787 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:34.103107    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:00:34.103787 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:34.103165    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3492be51-5e92-4369-8fc5-85d5ce0523fd-metrics-certs podName:3492be51-5e92-4369-8fc5-85d5ce0523fd nodeName:}" failed. No retries permitted until 2026-11-03 20:00:38.103146828 +0000 UTC m=+10.096303217 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3492be51-5e92-4369-8fc5-85d5ce0523fd-metrics-certs") pod "network-metrics-daemon-286gh" (UID: "3492be51-5e92-4369-8fc5-85d5ce0523fd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:00:34.103787 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:34.103266    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:00:34.103787 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:34.103283    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:00:34.103787 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:34.103298    2575 projected.go:196] Error preparing data for projected volume kube-api-access-dpxxg for pod openshift-network-diagnostics/network-check-target-h6crz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:00:34.103787 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:34.103336    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e8b3941-ab89-4838-bf55-6e9521ed259c-kube-api-access-dpxxg podName:6e8b3941-ab89-4838-bf55-6e9521ed259c nodeName:}" failed. No retries permitted until 2026-11-03 20:00:38.103324272 +0000 UTC m=+10.096480662 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dpxxg" (UniqueName: "kubernetes.io/projected/6e8b3941-ab89-4838-bf55-6e9521ed259c-kube-api-access-dpxxg") pod "network-check-target-h6crz" (UID: "6e8b3941-ab89-4838-bf55-6e9521ed259c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:00:34.518611 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:34.517120    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:00:34.518611 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:34.517321    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-286gh" podUID="3492be51-5e92-4369-8fc5-85d5ce0523fd"
Nov 03 20:00:34.581517 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:34.581486    2575 generic.go:356] "Generic (PLEG): container finished" podID="b45f8e0dedc76d41e693c990769490b8" containerID="9b5b24f3311434dd42b4ae883fca519bad8b0d0c1ab3dd8330cbc8471a674212" exitCode=0
Nov 03 20:00:34.581668 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:34.581546    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal" event={"ID":"b45f8e0dedc76d41e693c990769490b8","Type":"ContainerDied","Data":"9b5b24f3311434dd42b4ae883fca519bad8b0d0c1ab3dd8330cbc8471a674212"}
Nov 03 20:00:34.599482 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:34.599439    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal" podStartSLOduration=4.5994245209999995 podStartE2EDuration="4.599424521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:31.555180545 +0000 UTC m=+3.548336946" watchObservedRunningTime="2026-11-03 20:00:34.599424521 +0000 UTC m=+6.592580920"
Nov 03 20:00:35.517692 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:35.517652    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:35.517936 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:35.517804    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8wrn" podUID="0181bad8-3459-4c64-8e31-12395fd4c6c7"
Nov 03 20:00:35.518198 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:35.518182    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:00:35.518276 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:35.518259    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6crz" podUID="6e8b3941-ab89-4838-bf55-6e9521ed259c"
Nov 03 20:00:35.586346 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:35.586296    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal" event={"ID":"b45f8e0dedc76d41e693c990769490b8","Type":"ContainerStarted","Data":"140ab387404d67c43d205b26dd6f41bc2961ce4070715fb41eefd71e9362f005"}
Nov 03 20:00:35.593085 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:35.591942    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k5b5s" event={"ID":"07e7cf29-701a-4c9d-a3ad-12d4846157c2","Type":"ContainerStarted","Data":"c6f25867e80a23d56c39455c9c7fb492b7baafcd307aa69ac649fba579136164"}
Nov 03 20:00:35.626437 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:35.626383    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal" podStartSLOduration=5.626364426 podStartE2EDuration="5.626364426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:35.606197557 +0000 UTC m=+7.599353957" watchObservedRunningTime="2026-11-03 20:00:35.626364426 +0000 UTC m=+7.619520826"
Nov 03 20:00:35.626797 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:35.626750    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-k5b5s" podStartSLOduration=2.7510039920000002 podStartE2EDuration="6.626734481s" totalImagesPullingTime="3.875730489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:35.625877878 +0000 UTC m=+7.619034277" watchObservedRunningTime="2026-11-03 20:00:35.626734481 +0000 UTC m=+7.619890885"
Nov 03 20:00:36.519428 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:36.518888    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:00:36.519428 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:36.519048    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-286gh" podUID="3492be51-5e92-4369-8fc5-85d5ce0523fd"
Nov 03 20:00:36.597055 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:36.596957    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb" event={"ID":"71697994-0833-4570-83d3-afa5739725e9","Type":"ContainerStarted","Data":"855b1b4fc331b5211a2011946372b3e28023fb4cd5e1641ffd1b456a95721fbf"}
Nov 03 20:00:36.601452 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:36.600722    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tt4df" event={"ID":"78003043-36f1-452a-8ccf-37b574877715","Type":"ContainerStarted","Data":"2d01aa8831f162bc177c832d026dd11f34b323262053a2e18e578121f68a281a"}
Nov 03 20:00:36.606342 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:36.604129    2575 generic.go:356] "Generic (PLEG): container finished" podID="9d61c96a-2363-47b2-949a-300d9b084779" containerID="1bd35de346bfe6419bd34f96a66c09cb4b4cdece3bc45b90195becd4c71fc8cb" exitCode=0
Nov 03 20:00:36.606342 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:36.605141    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86vz2" event={"ID":"9d61c96a-2363-47b2-949a-300d9b084779","Type":"ContainerDied","Data":"1bd35de346bfe6419bd34f96a66c09cb4b4cdece3bc45b90195becd4c71fc8cb"}
Nov 03 20:00:36.645699 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:36.645541    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tt4df" podStartSLOduration=2.754971836 podStartE2EDuration="7.645525194s" totalImagesPullingTime="4.890553358s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:36.620232839 +0000 UTC m=+8.613389240" watchObservedRunningTime="2026-11-03 20:00:36.645525194 +0000 UTC m=+8.638681592"
Nov 03 20:00:37.518040 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:37.517992    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:00:37.518214 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:37.518167    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6crz" podUID="6e8b3941-ab89-4838-bf55-6e9521ed259c"
Nov 03 20:00:37.518460 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:37.518427    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:37.518588 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:37.518558    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8wrn" podUID="0181bad8-3459-4c64-8e31-12395fd4c6c7"
Nov 03 20:00:38.031997 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:38.031595    2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 20:00:38.142916 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:38.142834    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3492be51-5e92-4369-8fc5-85d5ce0523fd-metrics-certs\") pod \"network-metrics-daemon-286gh\" (UID: \"3492be51-5e92-4369-8fc5-85d5ce0523fd\") " pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:00:38.142916 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:38.142880    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxxg\" (UniqueName: \"kubernetes.io/projected/6e8b3941-ab89-4838-bf55-6e9521ed259c-kube-api-access-dpxxg\") pod \"network-check-target-h6crz\" (UID: \"6e8b3941-ab89-4838-bf55-6e9521ed259c\") " pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:00:38.142916 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:38.142903    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0181bad8-3459-4c64-8e31-12395fd4c6c7-original-pull-secret\") pod \"global-pull-secret-syncer-x8wrn\" (UID: \"0181bad8-3459-4c64-8e31-12395fd4c6c7\") " pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:38.143180 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:38.142999    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:00:38.143180 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:38.143014    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:00:38.143180 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:38.143044    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0181bad8-3459-4c64-8e31-12395fd4c6c7-original-pull-secret podName:0181bad8-3459-4c64-8e31-12395fd4c6c7 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:46.143031917 +0000 UTC m=+18.136188293 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0181bad8-3459-4c64-8e31-12395fd4c6c7-original-pull-secret") pod "global-pull-secret-syncer-x8wrn" (UID: "0181bad8-3459-4c64-8e31-12395fd4c6c7") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:00:38.143180 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:38.143077    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3492be51-5e92-4369-8fc5-85d5ce0523fd-metrics-certs podName:3492be51-5e92-4369-8fc5-85d5ce0523fd nodeName:}" failed. No retries permitted until 2026-11-03 20:00:46.143059637 +0000 UTC m=+18.136216026 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3492be51-5e92-4369-8fc5-85d5ce0523fd-metrics-certs") pod "network-metrics-daemon-286gh" (UID: "3492be51-5e92-4369-8fc5-85d5ce0523fd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:00:38.143180 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:38.143171    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:00:38.143407 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:38.143187    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:00:38.143407 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:38.143200    2575 projected.go:196] Error preparing data for projected volume kube-api-access-dpxxg for pod openshift-network-diagnostics/network-check-target-h6crz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:00:38.143407 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:38.143241    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e8b3941-ab89-4838-bf55-6e9521ed259c-kube-api-access-dpxxg podName:6e8b3941-ab89-4838-bf55-6e9521ed259c nodeName:}" failed. No retries permitted until 2026-11-03 20:00:46.143227926 +0000 UTC m=+18.136384308 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dpxxg" (UniqueName: "kubernetes.io/projected/6e8b3941-ab89-4838-bf55-6e9521ed259c-kube-api-access-dpxxg") pod "network-check-target-h6crz" (UID: "6e8b3941-ab89-4838-bf55-6e9521ed259c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:00:38.428008 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:38.427975    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-k5b5s"
Nov 03 20:00:38.428338 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:38.427915    2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:00:38.031612607Z","UUID":"6cf50a64-bb04-4c9d-9241-6322865c6e79","Handler":null,"Name":"","Endpoint":""}
Nov 03 20:00:38.429710 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:38.429010    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-k5b5s"
Nov 03 20:00:38.430434 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:38.430414    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 20:00:38.430542 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:38.430447    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 20:00:38.518491 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:38.518465    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:00:38.518665 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:38.518639    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-286gh" podUID="3492be51-5e92-4369-8fc5-85d5ce0523fd"
Nov 03 20:00:38.611476 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:38.611439    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mq5xz" event={"ID":"ac4c4fc1-a4c5-4f00-97bc-b201eede1226","Type":"ContainerStarted","Data":"cca680b5e03deb5acd2ef41522097d9205df2a6a36602d6d22733fd146adaf1b"}
Nov 03 20:00:38.614700 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:38.614615    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb" event={"ID":"71697994-0833-4570-83d3-afa5739725e9","Type":"ContainerStarted","Data":"e4a43284e26a9a535ed3553e5ce1de65a34cd3c09587e6db1d25bbd7b4b8e1a1"}
Nov 03 20:00:38.616609 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:38.616285    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g2rp5" event={"ID":"153d1ce3-22c2-444c-9bd2-ee91ff044d29","Type":"ContainerStarted","Data":"df5a7424fdb954147032e1ffefb3d3f1ffa092c937d593a68e4dec4bbc9f95e7"}
Nov 03 20:00:38.633736 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:38.633682    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-mq5xz" podStartSLOduration=2.535337006 podStartE2EDuration="9.633668995s" totalImagesPullingTime="7.098331989s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:38.633359229 +0000 UTC m=+10.626515626" watchObservedRunningTime="2026-11-03 20:00:38.633668995 +0000 UTC m=+10.626825392"
Nov 03 20:00:38.651866 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:38.651770    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-g2rp5" podStartSLOduration=1.990219784 podStartE2EDuration="9.651757441s" totalImagesPullingTime="7.661537657s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:38.651094653 +0000 UTC m=+10.644251054" watchObservedRunningTime="2026-11-03 20:00:38.651757441 +0000 UTC m=+10.644913842"
Nov 03 20:00:39.517722 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:39.517690    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:39.518097 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:39.517821    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8wrn" podUID="0181bad8-3459-4c64-8e31-12395fd4c6c7"
Nov 03 20:00:39.518181 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:39.518160    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:00:39.518284 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:39.518257    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6crz" podUID="6e8b3941-ab89-4838-bf55-6e9521ed259c"
Nov 03 20:00:39.620876 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:39.620800    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb" event={"ID":"71697994-0833-4570-83d3-afa5739725e9","Type":"ContainerStarted","Data":"a3f41c10d7f190c58492d0a431302eed4721e22aa444c38b8364eeab65dc8f13"}
Nov 03 20:00:39.623550 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:39.623494    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86vz2" event={"ID":"9d61c96a-2363-47b2-949a-300d9b084779","Type":"ContainerStarted","Data":"c37be76c398d4cbf1a38bc3024327db2064b247dc54f2fcfdb08d1476fa95252"}
Nov 03 20:00:39.672094 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:39.671157    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb" podStartSLOduration=2.684325597 podStartE2EDuration="10.671138891s" totalImagesPullingTime="7.986813294s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:39.644776603 +0000 UTC m=+11.637933003" watchObservedRunningTime="2026-11-03 20:00:39.671138891 +0000 UTC m=+11.664295293"
Nov 03 20:00:40.517329 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:40.517119    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:00:40.517511 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:40.517444    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-286gh" podUID="3492be51-5e92-4369-8fc5-85d5ce0523fd"
Nov 03 20:00:40.628423 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:40.628384    2575 generic.go:356] "Generic (PLEG): container finished" podID="9d61c96a-2363-47b2-949a-300d9b084779" containerID="c37be76c398d4cbf1a38bc3024327db2064b247dc54f2fcfdb08d1476fa95252" exitCode=0
Nov 03 20:00:40.628906 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:40.628474    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86vz2" event={"ID":"9d61c96a-2363-47b2-949a-300d9b084779","Type":"ContainerDied","Data":"c37be76c398d4cbf1a38bc3024327db2064b247dc54f2fcfdb08d1476fa95252"}
Nov 03 20:00:40.631746 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:40.631714    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v9qll" event={"ID":"c5965c2f-be74-431c-a8ff-86598070072a","Type":"ContainerStarted","Data":"8cdcf4dd03e46243d384a9a9f04b0fa1b0bbebcc52a5921addf5e25dabdeae94"}
Nov 03 20:00:40.671611 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:40.671549    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-v9qll" podStartSLOduration=4.042687351 podStartE2EDuration="11.671531595s" totalImagesPullingTime="7.628844244s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:40.671262728 +0000 UTC m=+12.664419129" watchObservedRunningTime="2026-11-03 20:00:40.671531595 +0000 UTC m=+12.664687994"
Nov 03 20:00:41.517127 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:41.517085    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:41.517322 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:41.517222    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8wrn" podUID="0181bad8-3459-4c64-8e31-12395fd4c6c7"
Nov 03 20:00:41.517600 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:41.517578    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:00:41.517728 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:41.517688    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6crz" podUID="6e8b3941-ab89-4838-bf55-6e9521ed259c"
Nov 03 20:00:42.519805 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:42.519741    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:00:42.520300 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:42.519839    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-286gh" podUID="3492be51-5e92-4369-8fc5-85d5ce0523fd"
Nov 03 20:00:42.638875 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:42.638834    2575 generic.go:356] "Generic (PLEG): container finished" podID="9d61c96a-2363-47b2-949a-300d9b084779" containerID="ed24c50e209a0962486fd8774e1154afef6119dfc7499c20853b0e8e3a8f1e38" exitCode=0
Nov 03 20:00:42.639040 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:42.638899    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86vz2" event={"ID":"9d61c96a-2363-47b2-949a-300d9b084779","Type":"ContainerDied","Data":"ed24c50e209a0962486fd8774e1154afef6119dfc7499c20853b0e8e3a8f1e38"}
Nov 03 20:00:43.517633 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:43.517584    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:43.517633 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:43.517626    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:00:43.517802 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:43.517714    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8wrn" podUID="0181bad8-3459-4c64-8e31-12395fd4c6c7"
Nov 03 20:00:43.517846 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:43.517834    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6crz" podUID="6e8b3941-ab89-4838-bf55-6e9521ed259c"
Nov 03 20:00:43.644011 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:43.643925    2575 generic.go:356] "Generic (PLEG): container finished" podID="9d61c96a-2363-47b2-949a-300d9b084779" containerID="3e6b28af71e66bdab796ea5999c446c83ca51b1361a583f82e8c53ab8abc64d1" exitCode=0
Nov 03 20:00:43.644011 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:43.643982    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86vz2" event={"ID":"9d61c96a-2363-47b2-949a-300d9b084779","Type":"ContainerDied","Data":"3e6b28af71e66bdab796ea5999c446c83ca51b1361a583f82e8c53ab8abc64d1"}
Nov 03 20:00:44.517154 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:44.517107    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:00:44.517343 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:44.517254    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-286gh" podUID="3492be51-5e92-4369-8fc5-85d5ce0523fd"
Nov 03 20:00:45.517977 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:45.517371    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:45.517977 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:45.517495    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8wrn" podUID="0181bad8-3459-4c64-8e31-12395fd4c6c7"
Nov 03 20:00:45.517977 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:45.517864    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:00:45.517977 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:45.517935    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6crz" podUID="6e8b3941-ab89-4838-bf55-6e9521ed259c"
Nov 03 20:00:46.208045 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:46.207998    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3492be51-5e92-4369-8fc5-85d5ce0523fd-metrics-certs\") pod \"network-metrics-daemon-286gh\" (UID: \"3492be51-5e92-4369-8fc5-85d5ce0523fd\") " pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:00:46.208045 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:46.208066    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxxg\" (UniqueName: \"kubernetes.io/projected/6e8b3941-ab89-4838-bf55-6e9521ed259c-kube-api-access-dpxxg\") pod \"network-check-target-h6crz\" (UID: \"6e8b3941-ab89-4838-bf55-6e9521ed259c\") " pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:00:46.208045 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:46.208097    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0181bad8-3459-4c64-8e31-12395fd4c6c7-original-pull-secret\") pod \"global-pull-secret-syncer-x8wrn\" (UID: \"0181bad8-3459-4c64-8e31-12395fd4c6c7\") " pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:46.208529 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:46.208233    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 20:00:46.208529 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:46.208288    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0181bad8-3459-4c64-8e31-12395fd4c6c7-original-pull-secret podName:0181bad8-3459-4c64-8e31-12395fd4c6c7 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:02.208269962 +0000 UTC m=+34.201426354 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0181bad8-3459-4c64-8e31-12395fd4c6c7-original-pull-secret") pod "global-pull-secret-syncer-x8wrn" (UID: "0181bad8-3459-4c64-8e31-12395fd4c6c7") : object "kube-system"/"original-pull-secret" not registered
Nov 03 20:00:46.208901 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:46.208684    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:00:46.208901 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:46.208742    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3492be51-5e92-4369-8fc5-85d5ce0523fd-metrics-certs podName:3492be51-5e92-4369-8fc5-85d5ce0523fd nodeName:}" failed. No retries permitted until 2026-11-03 20:01:02.208723351 +0000 UTC m=+34.201879738 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3492be51-5e92-4369-8fc5-85d5ce0523fd-metrics-certs") pod "network-metrics-daemon-286gh" (UID: "3492be51-5e92-4369-8fc5-85d5ce0523fd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:00:46.208901 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:46.208814    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 20:00:46.208901 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:46.208833    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 20:00:46.208901 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:46.208843    2575 projected.go:196] Error preparing data for projected volume kube-api-access-dpxxg for pod openshift-network-diagnostics/network-check-target-h6crz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:00:46.208901 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:46.208867    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6e8b3941-ab89-4838-bf55-6e9521ed259c-kube-api-access-dpxxg podName:6e8b3941-ab89-4838-bf55-6e9521ed259c nodeName:}" failed. No retries permitted until 2026-11-03 20:01:02.208860277 +0000 UTC m=+34.202016649 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dpxxg" (UniqueName: "kubernetes.io/projected/6e8b3941-ab89-4838-bf55-6e9521ed259c-kube-api-access-dpxxg") pod "network-check-target-h6crz" (UID: "6e8b3941-ab89-4838-bf55-6e9521ed259c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 20:00:46.518471 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:46.517998    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:00:46.518471 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:46.518134    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-286gh" podUID="3492be51-5e92-4369-8fc5-85d5ce0523fd"
Nov 03 20:00:46.655103 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:46.654252    2575 generic.go:356] "Generic (PLEG): container finished" podID="9d61c96a-2363-47b2-949a-300d9b084779" containerID="891162d75aebe49b104c7a6361b195048a38f8e912012c6e97df4f9800ce0377" exitCode=0
Nov 03 20:00:46.655103 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:46.654307    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86vz2" event={"ID":"9d61c96a-2363-47b2-949a-300d9b084779","Type":"ContainerDied","Data":"891162d75aebe49b104c7a6361b195048a38f8e912012c6e97df4f9800ce0377"}
Nov 03 20:00:47.518032 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:47.517809    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:00:47.518198 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:47.517834    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:47.518198 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:47.518128    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6crz" podUID="6e8b3941-ab89-4838-bf55-6e9521ed259c"
Nov 03 20:00:47.518309 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:47.518250    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8wrn" podUID="0181bad8-3459-4c64-8e31-12395fd4c6c7"
Nov 03 20:00:47.659257 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:47.659158    2575 generic.go:356] "Generic (PLEG): container finished" podID="9d61c96a-2363-47b2-949a-300d9b084779" containerID="715874c6b232170f5ec7d0d3ffbd1e914d9c8e6184cdc8f8961687c325d498c9" exitCode=0
Nov 03 20:00:47.659774 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:47.659253    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86vz2" event={"ID":"9d61c96a-2363-47b2-949a-300d9b084779","Type":"ContainerDied","Data":"715874c6b232170f5ec7d0d3ffbd1e914d9c8e6184cdc8f8961687c325d498c9"}
Nov 03 20:00:47.660534 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:47.660512    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w72hd" event={"ID":"368fd902-a222-4a5b-9bbf-a8decb45ad31","Type":"ContainerStarted","Data":"08a256848244e068549ee29a585bb4c1d6da5ea9b991cc149fd9cab4462defb7"}
Nov 03 20:00:47.710563 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:47.710523    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-w72hd" podStartSLOduration=2.145907297 podStartE2EDuration="18.710511397s" totalImagesPullingTime="16.5646041s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:47.710200506 +0000 UTC m=+19.703356903" watchObservedRunningTime="2026-11-03 20:00:47.710511397 +0000 UTC m=+19.703667792"
Nov 03 20:00:48.518029 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:48.517854    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:00:48.518208 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:48.518114    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-286gh" podUID="3492be51-5e92-4369-8fc5-85d5ce0523fd"
Nov 03 20:00:48.664793 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:48.664709    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r" event={"ID":"626900f3-1832-46a5-8c82-70d09284614c","Type":"ContainerStarted","Data":"b248268aedf14b17ff8db00870c48fe32f0c42ede00d9de072dfc79ca3272e56"}
Nov 03 20:00:48.665209 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:48.664802    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r" event={"ID":"626900f3-1832-46a5-8c82-70d09284614c","Type":"ContainerStarted","Data":"9ac611559966057bb13f239c1c9ab85acd6f280506bba641397158a591e1da87"}
Nov 03 20:00:48.665209 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:48.664841    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r" event={"ID":"626900f3-1832-46a5-8c82-70d09284614c","Type":"ContainerStarted","Data":"e240fb7efa85e291dd0c5337a5af8947f3f6fb6994d9f6f6faa60494b3b4db64"}
Nov 03 20:00:48.665209 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:48.664871    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r" event={"ID":"626900f3-1832-46a5-8c82-70d09284614c","Type":"ContainerStarted","Data":"70acbd905e16932f2c328dc534b8451638be38df6ae4734ae632db1cf1cb5bad"}
Nov 03 20:00:48.665209 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:48.664897    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r" event={"ID":"626900f3-1832-46a5-8c82-70d09284614c","Type":"ContainerStarted","Data":"7c83a84a0277070ee83ae179928f9603db63156e03ac67b204c9fe4b32a5184d"}
Nov 03 20:00:48.665209 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:48.664930    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r" event={"ID":"626900f3-1832-46a5-8c82-70d09284614c","Type":"ContainerStarted","Data":"4550bbaeed74f8e3f09e63a7710a8e8e9474b811c6c4e7af47a8efbe58e4a466"}
Nov 03 20:00:48.667796 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:48.667769    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-86vz2" event={"ID":"9d61c96a-2363-47b2-949a-300d9b084779","Type":"ContainerStarted","Data":"5137a51de3163368e48105b91d9f48593b93a35c19ed687a61eb38cbc1bda8d6"}
Nov 03 20:00:48.691960 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:48.691919    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-86vz2" podStartSLOduration=7.39686846 podStartE2EDuration="19.691909424s" totalImagesPullingTime="12.295040964s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:48.691626174 +0000 UTC m=+20.684782566" watchObservedRunningTime="2026-11-03 20:00:48.691909424 +0000 UTC m=+20.685065818"
Nov 03 20:00:49.517011 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:49.516978    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:49.517170 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:49.517096    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8wrn" podUID="0181bad8-3459-4c64-8e31-12395fd4c6c7"
Nov 03 20:00:49.517170 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:49.517155    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:00:49.517254 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:49.517236    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6crz" podUID="6e8b3941-ab89-4838-bf55-6e9521ed259c"
Nov 03 20:00:50.387707 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:50.387687    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-k5b5s"
Nov 03 20:00:50.388031 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:50.387806    2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 20:00:50.388254 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:50.388231    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k5b5s"
Nov 03 20:00:50.517476 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:50.517444    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:00:50.517658 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:50.517581    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-286gh" podUID="3492be51-5e92-4369-8fc5-85d5ce0523fd"
Nov 03 20:00:50.674801 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:50.674719    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r" event={"ID":"626900f3-1832-46a5-8c82-70d09284614c","Type":"ContainerStarted","Data":"db3a3326c68f37d1564cc1047a3fcd3b83be7f0e65ee498b6e9b391003a646a3"}
Nov 03 20:00:51.517152 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:51.517117    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:51.517591 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:51.517224    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8wrn" podUID="0181bad8-3459-4c64-8e31-12395fd4c6c7"
Nov 03 20:00:51.517591 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:51.517281    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:00:51.517591 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:51.517397    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6crz" podUID="6e8b3941-ab89-4838-bf55-6e9521ed259c"
Nov 03 20:00:52.517513 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:52.517487    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:00:52.517922 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:52.517660    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-286gh" podUID="3492be51-5e92-4369-8fc5-85d5ce0523fd"
Nov 03 20:00:52.682956 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:52.682921    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r" event={"ID":"626900f3-1832-46a5-8c82-70d09284614c","Type":"ContainerStarted","Data":"e00868be446a70af3bea9728a18140c42ce0a1cd876dd62b792288740eadfd72"}
Nov 03 20:00:52.683185 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:52.683158    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:52.698248 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:52.698227    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:52.715090 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:52.715026    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r" podStartSLOduration=6.415826514 podStartE2EDuration="23.715011445s" totalImagesPullingTime="17.299184931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:52.713472861 +0000 UTC m=+24.706629255" watchObservedRunningTime="2026-11-03 20:00:52.715011445 +0000 UTC m=+24.708167843"
Nov 03 20:00:53.517658 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:53.517633    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:00:53.517658 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:53.517633    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:53.518170 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:53.517737    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6crz" podUID="6e8b3941-ab89-4838-bf55-6e9521ed259c"
Nov 03 20:00:53.518170 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:53.517793    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8wrn" podUID="0181bad8-3459-4c64-8e31-12395fd4c6c7"
Nov 03 20:00:53.686035 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:53.685979    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:53.686187 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:53.686114    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:53.710355 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:53.710328    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:00:54.136964 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:54.136312    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x8wrn"]
Nov 03 20:00:54.136964 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:54.136485    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:54.136964 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:54.136626    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8wrn" podUID="0181bad8-3459-4c64-8e31-12395fd4c6c7"
Nov 03 20:00:54.137178 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:54.136990    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h6crz"]
Nov 03 20:00:54.137178 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:54.137087    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:00:54.137178 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:54.137166    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6crz" podUID="6e8b3941-ab89-4838-bf55-6e9521ed259c"
Nov 03 20:00:54.139050 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:54.139027    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-286gh"]
Nov 03 20:00:54.139161 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:54.139139    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:00:54.139263 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:54.139244    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-286gh" podUID="3492be51-5e92-4369-8fc5-85d5ce0523fd"
Nov 03 20:00:55.517275 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:55.517247    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:00:55.517801 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:55.517335    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6crz" podUID="6e8b3941-ab89-4838-bf55-6e9521ed259c"
Nov 03 20:00:55.517801 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:55.517405    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:55.517801 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:55.517488    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8wrn" podUID="0181bad8-3459-4c64-8e31-12395fd4c6c7"
Nov 03 20:00:56.518005 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:56.517804    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:00:56.518556 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:56.518120    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-286gh" podUID="3492be51-5e92-4369-8fc5-85d5ce0523fd"
Nov 03 20:00:57.517831 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:57.517797    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:00:57.517990 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:57.517797    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:57.517990 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:57.517915    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6crz" podUID="6e8b3941-ab89-4838-bf55-6e9521ed259c"
Nov 03 20:00:57.517990 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:57.517971    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8wrn" podUID="0181bad8-3459-4c64-8e31-12395fd4c6c7"
Nov 03 20:00:58.518382 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:58.518350    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:00:58.518776 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:58.518440    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-286gh" podUID="3492be51-5e92-4369-8fc5-85d5ce0523fd"
Nov 03 20:00:59.517476 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:59.517446    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:00:59.517671 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:00:59.517479    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:00:59.517671 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:59.517544    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h6crz" podUID="6e8b3941-ab89-4838-bf55-6e9521ed259c"
Nov 03 20:00:59.517768 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:00:59.517681    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8wrn" podUID="0181bad8-3459-4c64-8e31-12395fd4c6c7"
Nov 03 20:01:00.517661 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:00.517564    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:01:00.518111 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:01:00.517720    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-286gh" podUID="3492be51-5e92-4369-8fc5-85d5ce0523fd"
Nov 03 20:01:00.782253 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:00.782177    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-48.us-east-2.compute.internal" event="NodeReady"
Nov 03 20:01:00.782410 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:00.782296    2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 20:01:00.829805 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:00.828240    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wb9sn"]
Nov 03 20:01:00.917063 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:00.917021    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sx9rt"]
Nov 03 20:01:00.917234 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:00.917144    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:00.921046 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:00.921016    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 20:01:00.921166 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:00.921056    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 20:01:00.921166 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:00.921064    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7vkxg\""
Nov 03 20:01:00.931062 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:00.931044    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wb9sn"]
Nov 03 20:01:00.931131 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:00.931076    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sx9rt"]
Nov 03 20:01:00.931131 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:00.931091    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-74j6n"]
Nov 03 20:01:00.931212 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:00.931142    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sx9rt"
Nov 03 20:01:00.943163 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:00.943146    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-74j6n"]
Nov 03 20:01:00.943237 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:00.943231    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:01.011131 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.011105    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaaa35b2-aee2-40f8-956b-0b9f697dc994-tmp-dir\") pod \"dns-default-wb9sn\" (UID: \"eaaa35b2-aee2-40f8-956b-0b9f697dc994\") " pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:01.011131 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.011133    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-crio-socket\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:01.011287 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.011154    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:01.011287 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.011173    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/126ae513-97e3-4ef0-b415-5ec03ea09960-cert\") pod \"ingress-canary-sx9rt\" (UID: \"126ae513-97e3-4ef0-b415-5ec03ea09960\") " pod="openshift-ingress-canary/ingress-canary-sx9rt"
Nov 03 20:01:01.011287 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.011264    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzfmw\" (UniqueName: \"kubernetes.io/projected/eaaa35b2-aee2-40f8-956b-0b9f697dc994-kube-api-access-qzfmw\") pod \"dns-default-wb9sn\" (UID: \"eaaa35b2-aee2-40f8-956b-0b9f697dc994\") " pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:01.011287 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.011283    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-552wl\" (UniqueName: \"kubernetes.io/projected/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-kube-api-access-552wl\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:01.011404 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.011324    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n79bw\" (UniqueName: \"kubernetes.io/projected/126ae513-97e3-4ef0-b415-5ec03ea09960-kube-api-access-n79bw\") pod \"ingress-canary-sx9rt\" (UID: \"126ae513-97e3-4ef0-b415-5ec03ea09960\") " pod="openshift-ingress-canary/ingress-canary-sx9rt"
Nov 03 20:01:01.011404 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.011390    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-data-volume\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:01.011460 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.011406    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:01.011493 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.011466    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eaaa35b2-aee2-40f8-956b-0b9f697dc994-config-volume\") pod \"dns-default-wb9sn\" (UID: \"eaaa35b2-aee2-40f8-956b-0b9f697dc994\") " pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:01.011523 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.011491    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls\") pod \"dns-default-wb9sn\" (UID: \"eaaa35b2-aee2-40f8-956b-0b9f697dc994\") " pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:01.021766 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.021732    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 20:01:01.021937 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.021912    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 20:01:01.022051 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.021936    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 20:01:01.022051 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.021948    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 20:01:01.022051 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.021948    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 20:01:01.022051 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.021966    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 20:01:01.022051 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.021975    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 20:01:01.022051 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.021999    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-lrvk5\""
Nov 03 20:01:01.022051 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.022045    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9mszh\""
Nov 03 20:01:01.112393 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.112301    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n79bw\" (UniqueName: \"kubernetes.io/projected/126ae513-97e3-4ef0-b415-5ec03ea09960-kube-api-access-n79bw\") pod \"ingress-canary-sx9rt\" (UID: \"126ae513-97e3-4ef0-b415-5ec03ea09960\") " pod="openshift-ingress-canary/ingress-canary-sx9rt"
Nov 03 20:01:01.112393 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.112348    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-data-volume\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:01.112624 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.112473    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:01.112624 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.112531    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eaaa35b2-aee2-40f8-956b-0b9f697dc994-config-volume\") pod \"dns-default-wb9sn\" (UID: \"eaaa35b2-aee2-40f8-956b-0b9f697dc994\") " pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:01.112624 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.112553    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls\") pod \"dns-default-wb9sn\" (UID: \"eaaa35b2-aee2-40f8-956b-0b9f697dc994\") " pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:01.112624 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:01:01.112569    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 20:01:01.112624 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.112590    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaaa35b2-aee2-40f8-956b-0b9f697dc994-tmp-dir\") pod \"dns-default-wb9sn\" (UID: \"eaaa35b2-aee2-40f8-956b-0b9f697dc994\") " pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:01.112905 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.112635    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-crio-socket\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:01.112905 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:01:01.112658    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-insights-runtime-extractor-tls podName:20b2d254-c5b2-4a59-8ce9-49a9f2c0f435 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:01.612637159 +0000 UTC m=+33.605793532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-insights-runtime-extractor-tls") pod "insights-runtime-extractor-74j6n" (UID: "20b2d254-c5b2-4a59-8ce9-49a9f2c0f435") : secret "insights-runtime-extractor-tls" not found
Nov 03 20:01:01.112905 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.112697    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:01.112905 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.112701    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-data-volume\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:01.112905 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.112730    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/126ae513-97e3-4ef0-b415-5ec03ea09960-cert\") pod \"ingress-canary-sx9rt\" (UID: \"126ae513-97e3-4ef0-b415-5ec03ea09960\") " pod="openshift-ingress-canary/ingress-canary-sx9rt"
Nov 03 20:01:01.112905 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.112760    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzfmw\" (UniqueName: \"kubernetes.io/projected/eaaa35b2-aee2-40f8-956b-0b9f697dc994-kube-api-access-qzfmw\") pod \"dns-default-wb9sn\" (UID: \"eaaa35b2-aee2-40f8-956b-0b9f697dc994\") " pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:01.112905 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:01:01.112764    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 20:01:01.112905 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.112782    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-552wl\" (UniqueName: \"kubernetes.io/projected/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-kube-api-access-552wl\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:01.112905 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:01:01.112817    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls podName:eaaa35b2-aee2-40f8-956b-0b9f697dc994 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:01.612800833 +0000 UTC m=+33.605957211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls") pod "dns-default-wb9sn" (UID: "eaaa35b2-aee2-40f8-956b-0b9f697dc994") : secret "dns-default-metrics-tls" not found
Nov 03 20:01:01.112905 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.112821    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-crio-socket\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:01.113358 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.113099    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaaa35b2-aee2-40f8-956b-0b9f697dc994-tmp-dir\") pod \"dns-default-wb9sn\" (UID: \"eaaa35b2-aee2-40f8-956b-0b9f697dc994\") " pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:01.113358 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.113191    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eaaa35b2-aee2-40f8-956b-0b9f697dc994-config-volume\") pod \"dns-default-wb9sn\" (UID: \"eaaa35b2-aee2-40f8-956b-0b9f697dc994\") " pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:01.113358 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.113271    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:01.116398 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.116380    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/126ae513-97e3-4ef0-b415-5ec03ea09960-cert\") pod \"ingress-canary-sx9rt\" (UID: \"126ae513-97e3-4ef0-b415-5ec03ea09960\") " pod="openshift-ingress-canary/ingress-canary-sx9rt"
Nov 03 20:01:01.125189 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.125164    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n79bw\" (UniqueName: \"kubernetes.io/projected/126ae513-97e3-4ef0-b415-5ec03ea09960-kube-api-access-n79bw\") pod \"ingress-canary-sx9rt\" (UID: \"126ae513-97e3-4ef0-b415-5ec03ea09960\") " pod="openshift-ingress-canary/ingress-canary-sx9rt"
Nov 03 20:01:01.125267 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.125190    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-552wl\" (UniqueName: \"kubernetes.io/projected/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-kube-api-access-552wl\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:01.125267 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.125248    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzfmw\" (UniqueName: \"kubernetes.io/projected/eaaa35b2-aee2-40f8-956b-0b9f697dc994-kube-api-access-qzfmw\") pod \"dns-default-wb9sn\" (UID: \"eaaa35b2-aee2-40f8-956b-0b9f697dc994\") " pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:01.267483 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.267452    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sx9rt"
Nov 03 20:01:01.517833 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.517794    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:01:01.517833 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.517828    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:01:01.522143 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.522103    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 20:01:01.522143 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.522133    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 20:01:01.522314 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.522172    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 20:01:01.522314 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.522201    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nc2g5\""
Nov 03 20:01:01.522314 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.522288    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zc6h4\""
Nov 03 20:01:01.615135 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.615105    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sx9rt"]
Nov 03 20:01:01.616481 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.616467    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:01.616531 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.616498    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls\") pod \"dns-default-wb9sn\" (UID: \"eaaa35b2-aee2-40f8-956b-0b9f697dc994\") " pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:01.616622 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:01:01.616591    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 20:01:01.616669 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:01:01.616617    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 20:01:01.616669 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:01:01.616659    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls podName:eaaa35b2-aee2-40f8-956b-0b9f697dc994 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:02.616648245 +0000 UTC m=+34.609804617 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls") pod "dns-default-wb9sn" (UID: "eaaa35b2-aee2-40f8-956b-0b9f697dc994") : secret "dns-default-metrics-tls" not found
Nov 03 20:01:01.616742 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:01:01.616681    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-insights-runtime-extractor-tls podName:20b2d254-c5b2-4a59-8ce9-49a9f2c0f435 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:02.616667429 +0000 UTC m=+34.609823801 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-insights-runtime-extractor-tls") pod "insights-runtime-extractor-74j6n" (UID: "20b2d254-c5b2-4a59-8ce9-49a9f2c0f435") : secret "insights-runtime-extractor-tls" not found
Nov 03 20:01:01.618342 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:01:01.618320    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod126ae513_97e3_4ef0_b415_5ec03ea09960.slice/crio-5354369e587df64c92b52a4ef40995576fc57bab04f5b23aa525b810cc8751e9 WatchSource:0}: Error finding container 5354369e587df64c92b52a4ef40995576fc57bab04f5b23aa525b810cc8751e9: Status 404 returned error can't find the container with id 5354369e587df64c92b52a4ef40995576fc57bab04f5b23aa525b810cc8751e9
Nov 03 20:01:01.703291 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.703227    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sx9rt" event={"ID":"126ae513-97e3-4ef0-b415-5ec03ea09960","Type":"ContainerStarted","Data":"5354369e587df64c92b52a4ef40995576fc57bab04f5b23aa525b810cc8751e9"}
Nov 03 20:01:01.978808 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:01.978732    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g2rp5_153d1ce3-22c2-444c-9bd2-ee91ff044d29/dns-node-resolver/0.log"
Nov 03 20:01:02.220631 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.220582    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxxg\" (UniqueName: \"kubernetes.io/projected/6e8b3941-ab89-4838-bf55-6e9521ed259c-kube-api-access-dpxxg\") pod \"network-check-target-h6crz\" (UID: \"6e8b3941-ab89-4838-bf55-6e9521ed259c\") " pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:01:02.220788 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.220652    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0181bad8-3459-4c64-8e31-12395fd4c6c7-original-pull-secret\") pod \"global-pull-secret-syncer-x8wrn\" (UID: \"0181bad8-3459-4c64-8e31-12395fd4c6c7\") " pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:01:02.220788 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.220714    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3492be51-5e92-4369-8fc5-85d5ce0523fd-metrics-certs\") pod \"network-metrics-daemon-286gh\" (UID: \"3492be51-5e92-4369-8fc5-85d5ce0523fd\") " pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:01:02.220853 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:01:02.220820    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:01:02.220896 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:01:02.220886    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3492be51-5e92-4369-8fc5-85d5ce0523fd-metrics-certs podName:3492be51-5e92-4369-8fc5-85d5ce0523fd nodeName:}" failed. No retries permitted until 2026-11-03 20:01:34.22086814 +0000 UTC m=+66.214024514 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3492be51-5e92-4369-8fc5-85d5ce0523fd-metrics-certs") pod "network-metrics-daemon-286gh" (UID: "3492be51-5e92-4369-8fc5-85d5ce0523fd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 20:01:02.223756 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.223732    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0181bad8-3459-4c64-8e31-12395fd4c6c7-original-pull-secret\") pod \"global-pull-secret-syncer-x8wrn\" (UID: \"0181bad8-3459-4c64-8e31-12395fd4c6c7\") " pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:01:02.224083 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.224063    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpxxg\" (UniqueName: \"kubernetes.io/projected/6e8b3941-ab89-4838-bf55-6e9521ed259c-kube-api-access-dpxxg\") pod \"network-check-target-h6crz\" (UID: \"6e8b3941-ab89-4838-bf55-6e9521ed259c\") " pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:01:02.253856 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.253798    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-brm42"]
Nov 03 20:01:02.265914 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.265897    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-brm42"
Nov 03 20:01:02.267556 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.267535    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-brm42"]
Nov 03 20:01:02.273746 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.270262    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xg4bg\""
Nov 03 20:01:02.273746 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.272680    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 20:01:02.273746 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.272797    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 20:01:02.273746 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.272841    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 20:01:02.369905 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.369875    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 20:01:02.422148 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.422114    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/78748ccd-0718-470e-ade1-74ff636db6b7-signing-cabundle\") pod \"service-ca-5fffc97bdf-brm42\" (UID: \"78748ccd-0718-470e-ade1-74ff636db6b7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brm42"
Nov 03 20:01:02.422300 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.422170    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/78748ccd-0718-470e-ade1-74ff636db6b7-signing-key\") pod \"service-ca-5fffc97bdf-brm42\" (UID: \"78748ccd-0718-470e-ade1-74ff636db6b7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brm42"
Nov 03 20:01:02.422300 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.422193    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtcnf\" (UniqueName: \"kubernetes.io/projected/78748ccd-0718-470e-ade1-74ff636db6b7-kube-api-access-gtcnf\") pod \"service-ca-5fffc97bdf-brm42\" (UID: \"78748ccd-0718-470e-ade1-74ff636db6b7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brm42"
Nov 03 20:01:02.436165 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.436139    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:01:02.441413 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.441384    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:01:02.519120 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.518807    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:01:02.526310 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.523182    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/78748ccd-0718-470e-ade1-74ff636db6b7-signing-cabundle\") pod \"service-ca-5fffc97bdf-brm42\" (UID: \"78748ccd-0718-470e-ade1-74ff636db6b7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brm42"
Nov 03 20:01:02.526310 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.523257    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/78748ccd-0718-470e-ade1-74ff636db6b7-signing-key\") pod \"service-ca-5fffc97bdf-brm42\" (UID: \"78748ccd-0718-470e-ade1-74ff636db6b7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brm42"
Nov 03 20:01:02.526310 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.523287    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtcnf\" (UniqueName: \"kubernetes.io/projected/78748ccd-0718-470e-ade1-74ff636db6b7-kube-api-access-gtcnf\") pod \"service-ca-5fffc97bdf-brm42\" (UID: \"78748ccd-0718-470e-ade1-74ff636db6b7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brm42"
Nov 03 20:01:02.526310 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.524542    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/78748ccd-0718-470e-ade1-74ff636db6b7-signing-cabundle\") pod \"service-ca-5fffc97bdf-brm42\" (UID: \"78748ccd-0718-470e-ade1-74ff636db6b7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brm42"
Nov 03 20:01:02.526310 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.525888    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 20:01:02.526310 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.526044    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gz7vs\""
Nov 03 20:01:02.527897 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.527687    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/78748ccd-0718-470e-ade1-74ff636db6b7-signing-key\") pod \"service-ca-5fffc97bdf-brm42\" (UID: \"78748ccd-0718-470e-ade1-74ff636db6b7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brm42"
Nov 03 20:01:02.536652 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.536633    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtcnf\" (UniqueName: \"kubernetes.io/projected/78748ccd-0718-470e-ade1-74ff636db6b7-kube-api-access-gtcnf\") pod \"service-ca-5fffc97bdf-brm42\" (UID: \"78748ccd-0718-470e-ade1-74ff636db6b7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brm42"
Nov 03 20:01:02.593253 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.593217    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-brm42"
Nov 03 20:01:02.623864 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.623839    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:02.623976 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.623878    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls\") pod \"dns-default-wb9sn\" (UID: \"eaaa35b2-aee2-40f8-956b-0b9f697dc994\") " pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:02.624023 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:01:02.623996    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 20:01:02.624023 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:01:02.623996    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 20:01:02.624087 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:01:02.624063    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-insights-runtime-extractor-tls podName:20b2d254-c5b2-4a59-8ce9-49a9f2c0f435 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:04.62404462 +0000 UTC m=+36.617201006 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-insights-runtime-extractor-tls") pod "insights-runtime-extractor-74j6n" (UID: "20b2d254-c5b2-4a59-8ce9-49a9f2c0f435") : secret "insights-runtime-extractor-tls" not found
Nov 03 20:01:02.624087 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:01:02.624081    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls podName:eaaa35b2-aee2-40f8-956b-0b9f697dc994 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:04.624071831 +0000 UTC m=+36.617228206 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls") pod "dns-default-wb9sn" (UID: "eaaa35b2-aee2-40f8-956b-0b9f697dc994") : secret "dns-default-metrics-tls" not found
Nov 03 20:01:02.735579 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.735542    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h6crz"]
Nov 03 20:01:02.738621 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:01:02.738583    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e8b3941_ab89_4838_bf55_6e9521ed259c.slice/crio-8ec457fb89d4030aa2e44e14bf257664354820bff93167c2813636bc76df33cb WatchSource:0}: Error finding container 8ec457fb89d4030aa2e44e14bf257664354820bff93167c2813636bc76df33cb: Status 404 returned error can't find the container with id 8ec457fb89d4030aa2e44e14bf257664354820bff93167c2813636bc76df33cb
Nov 03 20:01:02.764443 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.764394    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x8wrn"]
Nov 03 20:01:02.767564 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:01:02.767542    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0181bad8_3459_4c64_8e31_12395fd4c6c7.slice/crio-4239b26cc8f0a599ac22e14a1a895b37df562b959267d15f1d470e8d0d12c484 WatchSource:0}: Error finding container 4239b26cc8f0a599ac22e14a1a895b37df562b959267d15f1d470e8d0d12c484: Status 404 returned error can't find the container with id 4239b26cc8f0a599ac22e14a1a895b37df562b959267d15f1d470e8d0d12c484
Nov 03 20:01:02.877136 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:02.877041    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-brm42"]
Nov 03 20:01:02.880453 ip-10-0-143-48 kubenswrapper[2575]: W1103 20:01:02.880428    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78748ccd_0718_470e_ade1_74ff636db6b7.slice/crio-7651c81fe1cd14150be05fdd1805ebdf3147180082460cc1554f5bcd9780afb7 WatchSource:0}: Error finding container 7651c81fe1cd14150be05fdd1805ebdf3147180082460cc1554f5bcd9780afb7: Status 404 returned error can't find the container with id 7651c81fe1cd14150be05fdd1805ebdf3147180082460cc1554f5bcd9780afb7
Nov 03 20:01:03.340017 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:03.339988    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tt4df_78003043-36f1-452a-8ccf-37b574877715/node-ca/0.log"
Nov 03 20:01:03.709672 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:03.709576    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h6crz" event={"ID":"6e8b3941-ab89-4838-bf55-6e9521ed259c","Type":"ContainerStarted","Data":"8ec457fb89d4030aa2e44e14bf257664354820bff93167c2813636bc76df33cb"}
Nov 03 20:01:03.711638 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:03.711551    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-brm42" event={"ID":"78748ccd-0718-470e-ade1-74ff636db6b7","Type":"ContainerStarted","Data":"7651c81fe1cd14150be05fdd1805ebdf3147180082460cc1554f5bcd9780afb7"}
Nov 03 20:01:03.713063 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:03.713035    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x8wrn" event={"ID":"0181bad8-3459-4c64-8e31-12395fd4c6c7","Type":"ContainerStarted","Data":"4239b26cc8f0a599ac22e14a1a895b37df562b959267d15f1d470e8d0d12c484"}
Nov 03 20:01:04.648877 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:04.648842    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:04.649068 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:04.648896    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls\") pod \"dns-default-wb9sn\" (UID: \"eaaa35b2-aee2-40f8-956b-0b9f697dc994\") " pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:04.649068 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:01:04.649006    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 20:01:04.649068 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:01:04.649066    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 20:01:04.649293 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:01:04.649081    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-insights-runtime-extractor-tls podName:20b2d254-c5b2-4a59-8ce9-49a9f2c0f435 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:08.649061326 +0000 UTC m=+40.642217719 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-insights-runtime-extractor-tls") pod "insights-runtime-extractor-74j6n" (UID: "20b2d254-c5b2-4a59-8ce9-49a9f2c0f435") : secret "insights-runtime-extractor-tls" not found
Nov 03 20:01:04.649293 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:01:04.649116    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls podName:eaaa35b2-aee2-40f8-956b-0b9f697dc994 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:08.649101363 +0000 UTC m=+40.642257752 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls") pod "dns-default-wb9sn" (UID: "eaaa35b2-aee2-40f8-956b-0b9f697dc994") : secret "dns-default-metrics-tls" not found
Nov 03 20:01:06.741227 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:06.741132    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sx9rt" event={"ID":"126ae513-97e3-4ef0-b415-5ec03ea09960","Type":"ContainerStarted","Data":"95a9282b2afdbe56b573f6854a6e3fccf7bd0480cda436edf96c3e25fbd4076c"}
Nov 03 20:01:06.769942 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:06.769885    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sx9rt" podStartSLOduration=1.913244299 podStartE2EDuration="6.769867354s" totalImagesPullingTime="4.856623055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:06.766546548 +0000 UTC m=+38.759702944" watchObservedRunningTime="2026-11-03 20:01:06.769867354 +0000 UTC m=+38.763023773"
Nov 03 20:01:07.745948 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:07.745576    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-brm42" event={"ID":"78748ccd-0718-470e-ade1-74ff636db6b7","Type":"ContainerStarted","Data":"84952d85261c3899a0a3343cd44e1595f68f1385cfe5db19dcf3d0479a02f462"}
Nov 03 20:01:07.748203 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:07.748175    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h6crz" event={"ID":"6e8b3941-ab89-4838-bf55-6e9521ed259c","Type":"ContainerStarted","Data":"3c7178aef4ae9852dcafcb1dfecebfc2fb88cf27d6ec25d9f103c512384f11fd"}
Nov 03 20:01:07.748349 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:07.748256    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:01:07.763251 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:07.763205    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-brm42" podStartSLOduration=1.185605384 podStartE2EDuration="5.763193086s" totalImagesPullingTime="4.577587702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:07.762577351 +0000 UTC m=+39.755733747" watchObservedRunningTime="2026-11-03 20:01:07.763193086 +0000 UTC m=+39.756349480"
Nov 03 20:01:07.780825 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:07.779897    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h6crz" podStartSLOduration=34.337658518 podStartE2EDuration="38.77988052s" totalImagesPullingTime="4.442222002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:07.779187058 +0000 UTC m=+39.772343452" watchObservedRunningTime="2026-11-03 20:01:07.77988052 +0000 UTC m=+39.773036916"
Nov 03 20:01:08.694647 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:08.694570    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:08.694647 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:08.694639    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls\") pod \"dns-default-wb9sn\" (UID: \"eaaa35b2-aee2-40f8-956b-0b9f697dc994\") " pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:08.694870 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:01:08.694728    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 20:01:08.694870 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:01:08.694780    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 20:01:08.694870 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:01:08.694793    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-insights-runtime-extractor-tls podName:20b2d254-c5b2-4a59-8ce9-49a9f2c0f435 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:16.694773801 +0000 UTC m=+48.687930194 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-insights-runtime-extractor-tls") pod "insights-runtime-extractor-74j6n" (UID: "20b2d254-c5b2-4a59-8ce9-49a9f2c0f435") : secret "insights-runtime-extractor-tls" not found
Nov 03 20:01:08.694870 ip-10-0-143-48 kubenswrapper[2575]: E1103 20:01:08.694821    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls podName:eaaa35b2-aee2-40f8-956b-0b9f697dc994 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:16.694812162 +0000 UTC m=+48.687968534 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls") pod "dns-default-wb9sn" (UID: "eaaa35b2-aee2-40f8-956b-0b9f697dc994") : secret "dns-default-metrics-tls" not found
Nov 03 20:01:09.753826 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:09.753785    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x8wrn" event={"ID":"0181bad8-3459-4c64-8e31-12395fd4c6c7","Type":"ContainerStarted","Data":"9bd20df984781a814e2a3f4f325fba9fd2e55debd2b6d3979c4ae3f7a2de6296"}
Nov 03 20:01:09.770510 ip-10-0-143-48 kubenswrapper[2575]: I1103 20:01:09.770426    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-x8wrn" podStartSLOduration=34.028016258 podStartE2EDuration="40.770413195s" totalImagesPullingTime="6.742396937s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:09.769738453 +0000 UTC m=+41.762894849" watchObservedRunningTime="2026-11-03 20:01:09.770413195 +0000 UTC m=+41.763569624"
Nov 03 20:01:09.813538 ip-10-0-143-48 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 20:01:09.827878 ip-10-0-143-48 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 20:01:09.828045 ip-10-0-143-48 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 20:01:09.828113 ip-10-0-143-48 systemd[1]: kubelet.service: Consumed 2.387s CPU time.
Nov 03 20:01:09.837050 ip-10-0-143-48 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 20:01:09.933285 ip-10-0-143-48 kubenswrapper[4421]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:01:09.933285 ip-10-0-143-48 kubenswrapper[4421]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 20:01:09.933285 ip-10-0-143-48 kubenswrapper[4421]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:01:09.933285 ip-10-0-143-48 kubenswrapper[4421]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 20:01:09.935238 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935160    4421 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:01:09.935238 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935237    4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:01:09.935238 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935241    4421 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:01:09.935346 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935244    4421 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:01:09.935346 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935247    4421 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:01:09.935346 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935250    4421 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:01:09.935346 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935253    4421 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:01:09.935346 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935255    4421 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:01:09.935346 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935258    4421 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:01:09.935346 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935261    4421 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:01:09.935346 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935264    4421 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:01:09.935346 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935266    4421 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:01:09.935346 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935269    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:01:09.935346 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935272    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:01:09.935346 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935276    4421 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:01:09.935346 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935280    4421 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:01:09.935346 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935283    4421 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:01:09.935346 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935285    4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:01:09.935346 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935290    4421 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:01:09.935346 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935294    4421 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:01:09.935346 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935297    4421 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:01:09.935346 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935308    4421 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:01:09.935846 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935311    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:01:09.935846 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935314    4421 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:01:09.935846 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935316    4421 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:01:09.935846 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935318    4421 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:01:09.935846 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935321    4421 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:01:09.935846 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935323    4421 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:01:09.935846 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935326    4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:01:09.935846 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935329    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:01:09.935846 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935332    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:01:09.935846 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935335    4421 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:01:09.935846 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935338    4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:01:09.935846 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935340    4421 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:01:09.935846 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935343    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:01:09.935846 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935345    4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:01:09.935846 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935348    4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:01:09.935846 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935351    4421 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:01:09.935846 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935354    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:01:09.935846 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935356    4421 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:01:09.935846 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935359    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:01:09.935846 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935367    4421 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:01:09.936337 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935370    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:01:09.936337 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935372    4421 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:01:09.936337 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935375    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:01:09.936337 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935378    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:01:09.936337 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935380    4421 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:01:09.936337 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935383    4421 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:01:09.936337 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935385    4421 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:01:09.936337 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935390    4421 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:01:09.936337 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935392    4421 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:01:09.936337 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935395    4421 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:01:09.936337 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935398    4421 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:01:09.936337 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935400    4421 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:01:09.936337 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935402    4421 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:01:09.936337 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935405    4421 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:01:09.936337 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935408    4421 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:01:09.936337 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935410    4421 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:01:09.936337 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935413    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:01:09.936337 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935416    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:01:09.936337 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935419    4421 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:01:09.936850 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935421    4421 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:01:09.936850 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935424    4421 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:01:09.936850 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935427    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:01:09.936850 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935429    4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:01:09.936850 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935432    4421 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:01:09.936850 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935434    4421 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:01:09.936850 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935437    4421 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:01:09.936850 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935440    4421 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:01:09.936850 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935442    4421 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:01:09.936850 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935445    4421 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:01:09.936850 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935447    4421 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:01:09.936850 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935449    4421 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:01:09.936850 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935452    4421 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:01:09.936850 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935454    4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:01:09.936850 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935457    4421 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:01:09.936850 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935460    4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:01:09.936850 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935463    4421 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:01:09.936850 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935465    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:01:09.936850 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935467    4421 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:01:09.936850 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935470    4421 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:01:09.937341 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935472    4421 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:01:09.937341 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935475    4421 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:01:09.937341 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935477    4421 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:01:09.937341 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935480    4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:01:09.937341 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935482    4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:01:09.937341 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935485    4421 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:01:09.937341 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935487    4421 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:01:09.937341 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935490    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:01:09.937341 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935492    4421 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:01:09.937341 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935495    4421 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:01:09.937341 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935497    4421 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:01:09.937341 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935500    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:01:09.937341 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935503    4421 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:01:09.937341 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935505    4421 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:01:09.937341 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935508    4421 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:01:09.937341 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935510    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:01:09.937341 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935513    4421 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:01:09.937341 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935515    4421 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:01:09.937341 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935517    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:01:09.937341 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935520    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:01:09.937867 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935522    4421 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:01:09.937867 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935525    4421 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:01:09.937867 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935527    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:01:09.937867 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935529    4421 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:01:09.937867 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.935532    4421 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:01:09.937867 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936276    4421 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:01:09.937867 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936283    4421 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:01:09.937867 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936286    4421 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:01:09.937867 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936288    4421 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:01:09.937867 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936293    4421 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:01:09.937867 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936297    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:01:09.937867 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936300    4421 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:01:09.937867 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936303    4421 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:01:09.937867 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936305    4421 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:01:09.937867 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936310    4421 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:01:09.937867 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936313    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:01:09.937867 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936316    4421 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:01:09.937867 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936319    4421 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:01:09.937867 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936321    4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:01:09.937867 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936324    4421 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:01:09.938364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936326    4421 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:01:09.938364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936329    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:01:09.938364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936332    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:01:09.938364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936334    4421 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:01:09.938364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936337    4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:01:09.938364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936339    4421 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:01:09.938364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936342    4421 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:01:09.938364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936345    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:01:09.938364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936347    4421 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:01:09.938364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936350    4421 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:01:09.938364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936352    4421 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:01:09.938364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936355    4421 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:01:09.938364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936357    4421 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:01:09.938364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936360    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:01:09.938364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936363    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:01:09.938364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936365    4421 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:01:09.938364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936367    4421 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:01:09.938364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936370    4421 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:01:09.938364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936372    4421 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:01:09.938364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936375    4421 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:01:09.938896 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936377    4421 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:01:09.938896 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936379    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:01:09.938896 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936382    4421 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:01:09.938896 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936384    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:01:09.938896 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936387    4421 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:01:09.938896 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936389    4421 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:01:09.938896 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936393    4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:01:09.938896 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936396    4421 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:01:09.938896 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936399    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:01:09.938896 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936403    4421 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:01:09.938896 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936407    4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:01:09.938896 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936409    4421 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:01:09.938896 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936412    4421 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:01:09.938896 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936415    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:01:09.938896 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936417    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:01:09.938896 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936419    4421 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:01:09.938896 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936422    4421 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:01:09.938896 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936424    4421 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:01:09.938896 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936427    4421 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:01:09.939368 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936430    4421 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:01:09.939368 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936432    4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:01:09.939368 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936435    4421 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:01:09.939368 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936437    4421 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:01:09.939368 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936439    4421 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:01:09.939368 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936442    4421 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:01:09.939368 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936444    4421 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:01:09.939368 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936447    4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:01:09.939368 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936449    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:01:09.939368 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936452    4421 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:01:09.939368 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936455    4421 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:01:09.939368 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936457    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:01:09.939368 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936459    4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:01:09.939368 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936462    4421 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:01:09.939368 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936465    4421 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:01:09.939368 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936467    4421 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:01:09.939368 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936470    4421 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:01:09.939368 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936473    4421 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:01:09.939368 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936475    4421 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:01:09.939368 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936478    4421 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:01:09.939873 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936481    4421 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:01:09.939873 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936484    4421 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:01:09.939873 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936486    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:01:09.939873 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936489    4421 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:01:09.939873 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936491    4421 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:01:09.939873 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936494    4421 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:01:09.939873 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936496    4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:01:09.939873 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936499    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:01:09.939873 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936501    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:01:09.939873 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936504    4421 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:01:09.939873 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936506    4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:01:09.939873 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936509    4421 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:01:09.939873 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936511    4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:01:09.939873 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936513    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:01:09.939873 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936516    4421 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:01:09.939873 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936518    4421 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:01:09.939873 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936521    4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:01:09.939873 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936523    4421 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:01:09.939873 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936525    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:01:09.940381 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936528    4421 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:01:09.940381 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936530    4421 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:01:09.940381 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936532    4421 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:01:09.940381 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936535    4421 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:01:09.940381 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936537    4421 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:01:09.940381 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936540    4421 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:01:09.940381 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936542    4421 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:01:09.940381 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936545    4421 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:01:09.940381 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936548    4421 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:01:09.940381 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936551    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:01:09.940381 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936553    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:01:09.940381 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936556    4421 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:01:09.940381 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.936559    4421 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:01:09.940381 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937081    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:01:09.940381 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937087    4421 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:01:09.940381 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937090    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:01:09.940381 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937093    4421 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:01:09.940381 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937095    4421 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:01:09.940381 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937098    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:01:09.940381 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937101    4421 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:01:09.940915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937103    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:01:09.940915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937106    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:01:09.940915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937108    4421 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:01:09.940915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937111    4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:01:09.940915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937114    4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:01:09.940915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937117    4421 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:01:09.940915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937119    4421 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:01:09.940915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937122    4421 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:01:09.940915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937124    4421 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:01:09.940915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937127    4421 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:01:09.940915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937129    4421 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:01:09.940915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937132    4421 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:01:09.940915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937134    4421 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:01:09.940915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937137    4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:01:09.940915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937140    4421 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:01:09.940915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937142    4421 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:01:09.940915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937145    4421 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:01:09.940915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937147    4421 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:01:09.940915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937150    4421 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:01:09.940915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937153    4421 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:01:09.941514 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937156    4421 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:01:09.941514 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937158    4421 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:01:09.941514 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937161    4421 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:01:09.941514 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937164    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:01:09.941514 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937166    4421 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:01:09.941514 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937169    4421 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:01:09.941514 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937171    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:01:09.941514 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937174    4421 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:01:09.941514 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937176    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:01:09.941514 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937179    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:01:09.941514 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937181    4421 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:01:09.941514 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937184    4421 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:01:09.941514 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937186    4421 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:01:09.941514 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937188    4421 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:01:09.941514 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937191    4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:01:09.941514 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937193    4421 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:01:09.941514 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937196    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:01:09.941514 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937198    4421 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:01:09.941514 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937201    4421 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:01:09.941514 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937203    4421 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:01:09.942323 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937206    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:01:09.942323 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937208    4421 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:01:09.942323 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937210    4421 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:01:09.942323 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937213    4421 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:01:09.942323 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937215    4421 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:01:09.942323 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937218    4421 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:01:09.942323 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937220    4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:01:09.942323 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937222    4421 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:01:09.942323 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937225    4421 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:01:09.942323 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937227    4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:01:09.942323 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937230    4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:01:09.942323 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937233    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:01:09.942323 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937236    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:01:09.942323 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937238    4421 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:01:09.942323 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937241    4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:01:09.942323 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937243    4421 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:01:09.942323 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937246    4421 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:01:09.942323 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937248    4421 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:01:09.942323 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937251    4421 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:01:09.942323 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937253    4421 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:01:09.942895 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937256    4421 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:01:09.942895 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937259    4421 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:01:09.942895 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937261    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:01:09.942895 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937263    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:01:09.942895 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937266    4421 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:01:09.942895 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937268    4421 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:01:09.942895 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937271    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:01:09.942895 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937274    4421 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:01:09.942895 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937276    4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:01:09.942895 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937279    4421 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:01:09.942895 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937281    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:01:09.942895 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937284    4421 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:01:09.942895 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937287    4421 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:01:09.942895 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937289    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:01:09.942895 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937291    4421 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:01:09.942895 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937294    4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:01:09.942895 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937296    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:01:09.942895 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937299    4421 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:01:09.942895 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937302    4421 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:01:09.943364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937305    4421 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:01:09.943364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937309    4421 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:01:09.943364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937313    4421 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:01:09.943364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937317    4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:01:09.943364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937319    4421 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:01:09.943364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937322    4421 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:01:09.943364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937325    4421 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:01:09.943364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937328    4421 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:01:09.943364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937331    4421 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:01:09.943364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937333    4421 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:01:09.943364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937335    4421 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:01:09.943364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937339    4421 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:01:09.943364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937343    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:01:09.943364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937347    4421 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:01:09.943364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937350    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:01:09.943364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937353    4421 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:01:09.943364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937355    4421 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:01:09.943364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937358    4421 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:01:09.943364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937360    4421 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:01:09.943844 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.937363    4421 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:01:09.943844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937453    4421 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 20:01:09.943844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937460    4421 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 20:01:09.943844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937468    4421 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 20:01:09.943844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937474    4421 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 20:01:09.943844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937481    4421 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 20:01:09.943844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937486    4421 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 20:01:09.943844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937493    4421 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 20:01:09.943844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937497    4421 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 20:01:09.943844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937501    4421 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 20:01:09.943844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937504    4421 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 20:01:09.943844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937507    4421 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 20:01:09.943844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937510    4421 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 20:01:09.943844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937513    4421 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 20:01:09.943844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937519    4421 flags.go:64] FLAG: --cgroup-root=""
Nov 03 20:01:09.943844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937521    4421 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 20:01:09.943844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937524    4421 flags.go:64] FLAG: --client-ca-file=""
Nov 03 20:01:09.943844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937527    4421 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 20:01:09.943844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937530    4421 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 20:01:09.943844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937534    4421 flags.go:64] FLAG: --cluster-domain=""
Nov 03 20:01:09.943844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937538    4421 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 20:01:09.943844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937541    4421 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 20:01:09.943844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937544    4421 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 20:01:09.943844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937548    4421 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 20:01:09.944409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937551    4421 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 20:01:09.944409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937554    4421 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 20:01:09.944409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937557    4421 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 20:01:09.944409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937560    4421 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 20:01:09.944409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937564    4421 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 20:01:09.944409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937567    4421 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 20:01:09.944409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937570    4421 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 20:01:09.944409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937573    4421 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 20:01:09.944409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937576    4421 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 20:01:09.944409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937580    4421 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 20:01:09.944409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937583    4421 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 20:01:09.944409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937586    4421 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 20:01:09.944409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937588    4421 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 20:01:09.944409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937591    4421 flags.go:64] FLAG: --enable-server="true"
Nov 03 20:01:09.944409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937611    4421 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 20:01:09.944409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937615    4421 flags.go:64] FLAG: --event-burst="100"
Nov 03 20:01:09.944409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937619    4421 flags.go:64] FLAG: --event-qps="50"
Nov 03 20:01:09.944409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937621    4421 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 20:01:09.944409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937624    4421 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 20:01:09.944409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937627    4421 flags.go:64] FLAG: --eviction-hard=""
Nov 03 20:01:09.944409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937632    4421 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 20:01:09.944409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937635    4421 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 20:01:09.944409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937639    4421 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 20:01:09.944409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937642    4421 flags.go:64] FLAG: --eviction-soft=""
Nov 03 20:01:09.944409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937645    4421 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 20:01:09.945049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937648    4421 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 20:01:09.945049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937651    4421 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 20:01:09.945049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937654    4421 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 20:01:09.945049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937657    4421 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 20:01:09.945049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937660    4421 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 20:01:09.945049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937663    4421 flags.go:64] FLAG: --feature-gates=""
Nov 03 20:01:09.945049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937667    4421 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 20:01:09.945049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937670    4421 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 20:01:09.945049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937673    4421 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 20:01:09.945049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937676    4421 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 20:01:09.945049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937679    4421 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 20:01:09.945049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937682    4421 flags.go:64] FLAG: --help="false"
Nov 03 20:01:09.945049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937688    4421 flags.go:64] FLAG: --hostname-override="ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:01:09.945049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937692    4421 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 20:01:09.945049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937695    4421 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 20:01:09.945049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937698    4421 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 20:01:09.945049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937701    4421 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 20:01:09.945049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937705    4421 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 20:01:09.945049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937708    4421 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 20:01:09.945049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937710    4421 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 20:01:09.945049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937713    4421 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 20:01:09.945049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937717    4421 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 20:01:09.945049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937720    4421 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 20:01:09.945588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937723    4421 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 20:01:09.945588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937726    4421 flags.go:64] FLAG: --kube-reserved=""
Nov 03 20:01:09.945588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937728    4421 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 20:01:09.945588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937731    4421 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 20:01:09.945588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937734    4421 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 20:01:09.945588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937737    4421 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 20:01:09.945588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937741    4421 flags.go:64] FLAG: --lock-file=""
Nov 03 20:01:09.945588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937744    4421 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 20:01:09.945588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937746    4421 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 20:01:09.945588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937756    4421 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 20:01:09.945588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937762    4421 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 20:01:09.945588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937765    4421 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 20:01:09.945588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937768    4421 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 20:01:09.945588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937770    4421 flags.go:64] FLAG: --logging-format="text"
Nov 03 20:01:09.945588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937773    4421 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 20:01:09.945588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937776    4421 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 20:01:09.945588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937779    4421 flags.go:64] FLAG: --manifest-url=""
Nov 03 20:01:09.945588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937782    4421 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 20:01:09.945588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937786    4421 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 20:01:09.945588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937789    4421 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 20:01:09.945588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937793    4421 flags.go:64] FLAG: --max-pods="110"
Nov 03 20:01:09.945588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937796    4421 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 20:01:09.945588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937799    4421 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 20:01:09.945588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937802    4421 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 20:01:09.945588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937805    4421 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 20:01:09.946291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937808    4421 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 20:01:09.946291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937811    4421 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 20:01:09.946291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937814    4421 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 20:01:09.946291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937820    4421 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 20:01:09.946291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937823    4421 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 20:01:09.946291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937828    4421 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 20:01:09.946291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937831    4421 flags.go:64] FLAG: --pod-cidr=""
Nov 03 20:01:09.946291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937833    4421 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 20:01:09.946291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937836    4421 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 20:01:09.946291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937839    4421 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 20:01:09.946291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937842    4421 flags.go:64] FLAG: --port="10250"
Nov 03 20:01:09.946291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937845    4421 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 20:01:09.946291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937847    4421 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0ae94d7e1e01fa8f2"
Nov 03 20:01:09.946291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937852    4421 flags.go:64] FLAG: --qos-reserved=""
Nov 03 20:01:09.946291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937855    4421 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 20:01:09.946291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937858    4421 flags.go:64] FLAG: --register-node="true"
Nov 03 20:01:09.946291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937861    4421 flags.go:64] FLAG: --register-with-taints=""
Nov 03 20:01:09.946291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937864    4421 flags.go:64] FLAG: --registry-burst="10"
Nov 03 20:01:09.946291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937867    4421 flags.go:64] FLAG: --registry-qps="5"
Nov 03 20:01:09.946291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937870    4421 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 20:01:09.946291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937873    4421 flags.go:64] FLAG: --reserved-memory=""
Nov 03 20:01:09.946291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937877    4421 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 20:01:09.946291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937880    4421 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 20:01:09.946291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937883    4421 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 20:01:09.946291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937886    4421 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 20:01:09.946291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937889    4421 flags.go:64] FLAG: --runonce="false"
Nov 03 20:01:09.947013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937892    4421 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 20:01:09.947013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937895    4421 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 20:01:09.947013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937898    4421 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 20:01:09.947013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937901    4421 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 20:01:09.947013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937904    4421 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 20:01:09.947013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937907    4421 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 20:01:09.947013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937910    4421 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 20:01:09.947013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937913    4421 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 20:01:09.947013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937916    4421 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 20:01:09.947013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937919    4421 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 20:01:09.947013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937921    4421 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 20:01:09.947013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937925    4421 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 20:01:09.947013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937928    4421 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 20:01:09.947013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937931    4421 flags.go:64] FLAG: --system-cgroups=""
Nov 03 20:01:09.947013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937933    4421 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 20:01:09.947013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937939    4421 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 20:01:09.947013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937941    4421 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 20:01:09.947013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937944    4421 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 20:01:09.947013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937948    4421 flags.go:64] FLAG: --tls-min-version=""
Nov 03 20:01:09.947013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937951    4421 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 20:01:09.947013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937954    4421 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 20:01:09.947013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937957    4421 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 20:01:09.947013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937959    4421 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 20:01:09.947013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937962    4421 flags.go:64] FLAG: --v="2"
Nov 03 20:01:09.947013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937967    4421 flags.go:64] FLAG: --version="false"
Nov 03 20:01:09.947625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937971    4421 flags.go:64] FLAG: --vmodule=""
Nov 03 20:01:09.947625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937975    4421 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 20:01:09.947625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.937978    4421 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 20:01:09.947625 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938162    4421 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:01:09.947625 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938166    4421 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:01:09.947625 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938170    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:01:09.947625 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938173    4421 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:01:09.947625 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938175    4421 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:01:09.947625 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938178    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:01:09.947625 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938180    4421 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:01:09.947625 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938183    4421 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:01:09.947625 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938186    4421 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:01:09.947625 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938189    4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:01:09.947625 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938192    4421 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:01:09.947625 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938194    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:01:09.947625 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938197    4421 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:01:09.947625 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938200    4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:01:09.947625 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938202    4421 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:01:09.947625 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938205    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:01:09.947625 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938207    4421 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:01:09.948148 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938209    4421 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:01:09.948148 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938212    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:01:09.948148 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938214    4421 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:01:09.948148 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938217    4421 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:01:09.948148 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938219    4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:01:09.948148 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938221    4421 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:01:09.948148 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938224    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:01:09.948148 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938226    4421 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:01:09.948148 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938228    4421 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:01:09.948148 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938231    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:01:09.948148 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938233    4421 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:01:09.948148 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938236    4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:01:09.948148 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938238    4421 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:01:09.948148 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938240    4421 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:01:09.948148 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938243    4421 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:01:09.948148 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938245    4421 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:01:09.948148 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938248    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:01:09.948148 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938250    4421 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:01:09.948148 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938253    4421 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:01:09.948691 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938255    4421 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:01:09.948691 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938258    4421 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:01:09.948691 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938260    4421 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:01:09.948691 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938262    4421 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:01:09.948691 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938265    4421 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:01:09.948691 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938267    4421 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:01:09.948691 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938270    4421 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:01:09.948691 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938272    4421 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:01:09.948691 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938275    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:01:09.948691 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938277    4421 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:01:09.948691 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938280    4421 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:01:09.948691 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938283    4421 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:01:09.948691 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938285    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:01:09.948691 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938287    4421 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:01:09.948691 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938290    4421 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:01:09.948691 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938292    4421 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:01:09.948691 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938295    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:01:09.948691 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938297    4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:01:09.948691 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938299    4421 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:01:09.948691 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938301    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:01:09.949303 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938304    4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:01:09.949303 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938307    4421 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:01:09.949303 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938309    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:01:09.949303 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938311    4421 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:01:09.949303 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938314    4421 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:01:09.949303 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938316    4421 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:01:09.949303 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938319    4421 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:01:09.949303 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938321    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:01:09.949303 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938323    4421 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:01:09.949303 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938326    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:01:09.949303 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938328    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:01:09.949303 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938330    4421 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:01:09.949303 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938333    4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:01:09.949303 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938335    4421 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:01:09.949303 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938337    4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:01:09.949303 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938340    4421 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:01:09.949303 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938342    4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:01:09.949303 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938345    4421 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:01:09.949303 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938347    4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:01:09.949303 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938350    4421 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:01:09.949977 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938353    4421 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:01:09.949977 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938355    4421 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:01:09.949977 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938359    4421 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:01:09.949977 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938363    4421 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:01:09.949977 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938366    4421 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:01:09.949977 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938369    4421 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:01:09.949977 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938372    4421 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:01:09.949977 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938375    4421 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:01:09.949977 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938377    4421 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:01:09.949977 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938380    4421 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:01:09.949977 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938382    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:01:09.949977 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938385    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:01:09.949977 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938387    4421 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:01:09.949977 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938389    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:01:09.949977 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938392    4421 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:01:09.949977 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938394    4421 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:01:09.949977 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938397    4421 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:01:09.949977 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938399    4421 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:01:09.949977 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938403    4421 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:01:09.949977 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938406    4421 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:01:09.950524 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938408    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:01:09.950524 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938412    4421 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:01:09.950524 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938415    4421 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:01:09.950524 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938418    4421 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:01:09.950524 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938421    4421 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:01:09.950524 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938423    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:01:09.950524 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938427    4421 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:01:09.950524 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938430    4421 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:01:09.950524 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938432    4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:01:09.950524 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938435    4421 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:01:09.950524 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.938461    4421 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:01:09.950524 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938503    4421 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:01:09.950524 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938508    4421 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:01:09.950524 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938510    4421 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:01:09.950524 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938513    4421 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:01:09.950979 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938515    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:01:09.950979 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938518    4421 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:01:09.950979 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938521    4421 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:01:09.950979 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938523    4421 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:01:09.950979 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938526    4421 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:01:09.950979 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938529    4421 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:01:09.950979 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938531    4421 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:01:09.950979 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938534    4421 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:01:09.950979 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938536    4421 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:01:09.950979 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938539    4421 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:01:09.950979 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938541    4421 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:01:09.950979 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938544    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:01:09.950979 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938546    4421 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:01:09.950979 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938549    4421 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:01:09.950979 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938551    4421 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:01:09.950979 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938554    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:01:09.950979 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938558    4421 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:01:09.950979 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938560    4421 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:01:09.950979 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938563    4421 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:01:09.950979 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938565    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:01:09.951517 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938567    4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:01:09.951517 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938570    4421 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:01:09.951517 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938572    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:01:09.951517 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938576    4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:01:09.951517 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938579    4421 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:01:09.951517 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938581    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:01:09.951517 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938584    4421 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:01:09.951517 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938586    4421 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:01:09.951517 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938589    4421 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:01:09.951517 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938591    4421 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:01:09.951517 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938607    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:01:09.951517 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938610    4421 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:01:09.951517 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938612    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:01:09.951517 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938615    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:01:09.951517 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938617    4421 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:01:09.951517 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938620    4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:01:09.951517 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938622    4421 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:01:09.951517 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938625    4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:01:09.951517 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938627    4421 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:01:09.952288 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938630    4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:01:09.952288 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938632    4421 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:01:09.952288 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938634    4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:01:09.952288 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938637    4421 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:01:09.952288 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938639    4421 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:01:09.952288 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938642    4421 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:01:09.952288 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938646    4421 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:01:09.952288 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938649    4421 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:01:09.952288 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938652    4421 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:01:09.952288 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938656    4421 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:01:09.952288 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938659    4421 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:01:09.952288 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938662    4421 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:01:09.952288 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938665    4421 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:01:09.952288 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938667    4421 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:01:09.952288 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938670    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:01:09.952288 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938672    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:01:09.952288 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938676    4421 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:01:09.952288 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938678    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:01:09.952288 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938681    4421 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:01:09.952288 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938683    4421 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:01:09.953276 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938686    4421 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:01:09.953276 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938688    4421 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:01:09.953276 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938691    4421 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:01:09.953276 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938693    4421 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:01:09.953276 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938696    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:01:09.953276 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938700    4421 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:01:09.953276 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938708    4421 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:01:09.953276 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938711    4421 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:01:09.953276 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938713    4421 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:01:09.953276 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938716    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:01:09.953276 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938718    4421 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:01:09.953276 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938721    4421 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:01:09.953276 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938724    4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:01:09.953276 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938726    4421 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:01:09.953276 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938729    4421 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:01:09.953276 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938731    4421 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:01:09.953276 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938734    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:01:09.953276 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938736    4421 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:01:09.953276 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938739    4421 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:01:09.953276 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938741    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:01:09.954112 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938743    4421 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:01:09.954112 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938747    4421 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:01:09.954112 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938750    4421 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:01:09.954112 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938752    4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:01:09.954112 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938755    4421 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:01:09.954112 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938757    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:01:09.954112 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938760    4421 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:01:09.954112 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938764    4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:01:09.954112 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938768    4421 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:01:09.954112 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938770    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:01:09.954112 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938773    4421 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:01:09.954112 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938775    4421 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:01:09.954112 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938778    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:01:09.954112 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938780    4421 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:01:09.954112 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938783    4421 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:01:09.954112 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938785    4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:01:09.954112 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938787    4421 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:01:09.954112 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938790    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:01:09.954112 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938793    4421 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:01:09.954915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938796    4421 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:01:09.954915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938798    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:01:09.954915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938801    4421 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:01:09.954915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.938803    4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:01:09.954915 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.943566    4421 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 20:01:09.954915 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.943581    4421 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 20:01:09.954915 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.943615    4421 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 20:01:09.954915 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.943621    4421 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 20:01:09.954915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943678    4421 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:01:09.954915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943683    4421 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:01:09.954915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943686    4421 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:01:09.954915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943690    4421 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:01:09.954915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943693    4421 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:01:09.954915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943696    4421 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:01:09.954915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943699    4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:01:09.954915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943702    4421 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:01:09.954915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943704    4421 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:01:09.954915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943707    4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:01:09.954915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943709    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:01:09.955774 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943713    4421 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:01:09.955774 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943716    4421 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:01:09.955774 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943720    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:01:09.955774 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943724    4421 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:01:09.955774 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943726    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:01:09.955774 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943729    4421 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:01:09.955774 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943732    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:01:09.955774 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943735    4421 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:01:09.955774 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943737    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:01:09.955774 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943740    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:01:09.955774 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943742    4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:01:09.955774 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943744    4421 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:01:09.955774 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943747    4421 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:01:09.955774 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943749    4421 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:01:09.955774 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943752    4421 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:01:09.955774 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943754    4421 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:01:09.955774 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943756    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:01:09.955774 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943759    4421 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:01:09.955774 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943762    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:01:09.956571 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943764    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:01:09.956571 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943766    4421 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:01:09.956571 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943769    4421 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:01:09.956571 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943771    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:01:09.956571 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943774    4421 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:01:09.956571 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943776    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:01:09.956571 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943779    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:01:09.956571 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943781    4421 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:01:09.956571 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943783    4421 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:01:09.956571 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943786    4421 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:01:09.956571 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943788    4421 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:01:09.956571 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943791    4421 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:01:09.956571 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943793    4421 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:01:09.956571 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943796    4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:01:09.956571 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943798    4421 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:01:09.956571 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943801    4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:01:09.956571 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943811    4421 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:01:09.956571 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943814    4421 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:01:09.956571 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943816    4421 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:01:09.957089 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943819    4421 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:01:09.957089 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943821    4421 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:01:09.957089 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943824    4421 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:01:09.957089 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943828    4421 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:01:09.957089 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943832    4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:01:09.957089 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943835    4421 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:01:09.957089 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943838    4421 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:01:09.957089 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943840    4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:01:09.957089 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943843    4421 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:01:09.957089 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943845    4421 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:01:09.957089 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943848    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:01:09.957089 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943851    4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:01:09.957089 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943853    4421 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:01:09.957089 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943856    4421 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:01:09.957089 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943859    4421 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:01:09.957089 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943861    4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:01:09.957089 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943864    4421 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:01:09.957089 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943866    4421 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:01:09.957089 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943868    4421 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:01:09.957572 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943871    4421 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:01:09.957572 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943873    4421 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:01:09.957572 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943876    4421 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:01:09.957572 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943878    4421 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:01:09.957572 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943881    4421 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:01:09.957572 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943883    4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:01:09.957572 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943886    4421 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:01:09.957572 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943889    4421 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:01:09.957572 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943891    4421 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:01:09.957572 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943893    4421 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:01:09.957572 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943896    4421 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:01:09.957572 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943904    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:01:09.957572 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943907    4421 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:01:09.957572 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943909    4421 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:01:09.957572 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943911    4421 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:01:09.957572 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943914    4421 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:01:09.957572 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943917    4421 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:01:09.957572 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943919    4421 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:01:09.957572 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943922    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:01:09.958235 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943924    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:01:09.958235 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943927    4421 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:01:09.958235 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943929    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:01:09.958235 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943932    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:01:09.958235 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943934    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:01:09.958235 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943936    4421 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:01:09.958235 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943939    4421 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:01:09.958235 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943941    4421 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:01:09.958235 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943944    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:01:09.958235 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943946    4421 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:01:09.958235 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943949    4421 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:01:09.958235 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943951    4421 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:01:09.958235 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943953    4421 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:01:09.958235 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943956    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:01:09.958235 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943959    4421 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:01:09.958235 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943961    4421 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:01:09.958235 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943963    4421 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:01:09.958235 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943966    4421 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:01:09.958235 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.943968    4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:01:09.959019 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.943995    4421 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:01:09.959019 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944145    4421 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:01:09.959019 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944150    4421 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:01:09.959019 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944153    4421 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:01:09.959019 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944156    4421 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:01:09.959019 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944159    4421 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:01:09.959019 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944167    4421 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:01:09.959019 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944170    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:01:09.959019 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944172    4421 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:01:09.959019 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944175    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:01:09.959019 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944178    4421 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:01:09.959019 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944180    4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:01:09.959019 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944183    4421 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:01:09.959019 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944185    4421 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:01:09.959019 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944188    4421 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:01:09.959518 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944190    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:01:09.959518 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944192    4421 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:01:09.959518 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944195    4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:01:09.959518 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944197    4421 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:01:09.959518 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944199    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:01:09.959518 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944203    4421 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:01:09.959518 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944206    4421 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:01:09.959518 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944209    4421 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:01:09.959518 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944211    4421 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:01:09.959518 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944218    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:01:09.959518 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944221    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:01:09.959518 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944223    4421 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:01:09.959518 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944226    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:01:09.959518 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944228    4421 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:01:09.959518 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944231    4421 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:01:09.959518 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944235    4421 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:01:09.959518 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944237    4421 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:01:09.959518 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944239    4421 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:01:09.959518 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944242    4421 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:01:09.959518 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944244    4421 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:01:09.960072 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944247    4421 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:01:09.960072 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944249    4421 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:01:09.960072 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944253    4421 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:01:09.960072 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944256    4421 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:01:09.960072 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944261    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:01:09.960072 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944264    4421 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:01:09.960072 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944266    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:01:09.960072 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944269    4421 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:01:09.960072 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944271    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:01:09.960072 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944274    4421 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:01:09.960072 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944276    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:01:09.960072 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944278    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:01:09.960072 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944281    4421 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:01:09.960072 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944283    4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:01:09.960072 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944286    4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:01:09.960072 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944288    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:01:09.960072 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944291    4421 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:01:09.960072 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944293    4421 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:01:09.960072 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944296    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:01:09.960072 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944298    4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:01:09.960636 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944300    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:01:09.960636 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944304    4421 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:01:09.960636 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944307    4421 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:01:09.960636 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944310    4421 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:01:09.960636 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944312    4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:01:09.960636 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944314    4421 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:01:09.960636 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944317    4421 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:01:09.960636 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944321    4421 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:01:09.960636 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944323    4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:01:09.960636 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944325    4421 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:01:09.960636 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944328    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:01:09.960636 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944330    4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:01:09.960636 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944333    4421 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:01:09.960636 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944335    4421 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:01:09.960636 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944338    4421 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:01:09.960636 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944340    4421 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:01:09.960636 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944343    4421 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:01:09.960636 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944346    4421 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:01:09.960636 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944348    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:01:09.960636 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944351    4421 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:01:09.961189 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944353    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:01:09.961189 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944355    4421 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:01:09.961189 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944358    4421 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:01:09.961189 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944360    4421 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:01:09.961189 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944362    4421 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:01:09.961189 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944365    4421 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:01:09.961189 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944367    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:01:09.961189 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944370    4421 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:01:09.961189 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944372    4421 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:01:09.961189 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944374    4421 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:01:09.961189 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944377    4421 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:01:09.961189 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944379    4421 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:01:09.961189 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944382    4421 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:01:09.961189 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944386    4421 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:01:09.961189 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944388    4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:01:09.961189 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944391    4421 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:01:09.961189 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944393    4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:01:09.961189 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944396    4421 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:01:09.961189 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944398    4421 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:01:09.961189 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944402    4421 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:01:09.961822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944404    4421 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:01:09.961822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944407    4421 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:01:09.961822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944409    4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:01:09.961822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944412    4421 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:01:09.961822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944415    4421 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:01:09.961822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944417    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:01:09.961822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944420    4421 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:01:09.961822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944423    4421 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:01:09.961822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944425    4421 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:01:09.961822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944428    4421 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:01:09.961822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944431    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:01:09.961822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944433    4421 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:01:09.961822 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.944456    4421 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 20:01:09.961822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944489    4421 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 20:01:09.961822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944492    4421 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 20:01:09.961822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944495    4421 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 20:01:09.962258 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944497    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 20:01:09.962258 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944500    4421 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 20:01:09.962258 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944502    4421 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 20:01:09.962258 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944505    4421 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 20:01:09.962258 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944507    4421 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 20:01:09.962258 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944510    4421 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 20:01:09.962258 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944512    4421 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 20:01:09.962258 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944515    4421 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 20:01:09.962258 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944517    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 20:01:09.962258 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944520    4421 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 20:01:09.962258 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944523    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 20:01:09.962258 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944525    4421 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 20:01:09.962258 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944528    4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 20:01:09.962258 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944530    4421 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 20:01:09.962258 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944533    4421 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 20:01:09.962258 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944536    4421 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 20:01:09.962258 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944539    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 20:01:09.962258 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944541    4421 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 20:01:09.962258 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944543    4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 20:01:09.962822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944546    4421 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 20:01:09.962822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944548    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 20:01:09.962822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944552    4421 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 20:01:09.962822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944555    4421 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 20:01:09.962822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944557    4421 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 20:01:09.962822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944560    4421 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 20:01:09.962822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944563    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 20:01:09.962822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944565    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 20:01:09.962822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944568    4421 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 20:01:09.962822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944570    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 20:01:09.962822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944573    4421 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 20:01:09.962822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944575    4421 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 20:01:09.962822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944578    4421 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 20:01:09.962822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944580    4421 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 20:01:09.962822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944582    4421 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 20:01:09.962822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944585    4421 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 20:01:09.962822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944587    4421 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 20:01:09.962822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944589    4421 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 20:01:09.962822 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944592    4421 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 20:01:09.963364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944608    4421 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 20:01:09.963364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944612    4421 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 20:01:09.963364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944615    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 20:01:09.963364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944618    4421 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 20:01:09.963364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944621    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 20:01:09.963364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944623    4421 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 20:01:09.963364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944626    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 20:01:09.963364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944628    4421 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 20:01:09.963364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944631    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 20:01:09.963364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944634    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 20:01:09.963364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944636    4421 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 20:01:09.963364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944638    4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 20:01:09.963364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944641    4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 20:01:09.963364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944644    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 20:01:09.963364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944646    4421 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 20:01:09.963364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944649    4421 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 20:01:09.963364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944651    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 20:01:09.963364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944653    4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 20:01:09.963364 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944656    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 20:01:09.964001 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944658    4421 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 20:01:09.964001 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944661    4421 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 20:01:09.964001 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944664    4421 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 20:01:09.964001 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944667    4421 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 20:01:09.964001 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944669    4421 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 20:01:09.964001 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944672    4421 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 20:01:09.964001 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944674    4421 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 20:01:09.964001 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944676    4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 20:01:09.964001 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944679    4421 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 20:01:09.964001 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944681    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 20:01:09.964001 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944684    4421 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 20:01:09.964001 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944686    4421 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 20:01:09.964001 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944688    4421 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 20:01:09.964001 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944691    4421 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 20:01:09.964001 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944693    4421 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 20:01:09.964001 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944696    4421 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 20:01:09.964001 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944698    4421 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 20:01:09.964001 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944701    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 20:01:09.964001 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944703    4421 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 20:01:09.964001 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944706    4421 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 20:01:09.964866 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944708    4421 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 20:01:09.964866 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944711    4421 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 20:01:09.964866 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944713    4421 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 20:01:09.964866 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944716    4421 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 20:01:09.964866 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944718    4421 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 20:01:09.964866 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944721    4421 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 20:01:09.964866 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944723    4421 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 20:01:09.964866 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944725    4421 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 20:01:09.964866 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944728    4421 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 20:01:09.964866 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944730    4421 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 20:01:09.964866 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944733    4421 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 20:01:09.964866 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944735    4421 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 20:01:09.964866 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944737    4421 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 20:01:09.964866 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944740    4421 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 20:01:09.964866 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944742    4421 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 20:01:09.964866 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944745    4421 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 20:01:09.964866 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944747    4421 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 20:01:09.964866 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944750    4421 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 20:01:09.964866 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944752    4421 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 20:01:09.964866 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944755    4421 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 20:01:09.965855 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944757    4421 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 20:01:09.965855 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944759    4421 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 20:01:09.965855 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944762    4421 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 20:01:09.965855 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944766    4421 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 20:01:09.965855 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944768    4421 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 20:01:09.965855 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:09.944771    4421 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 20:01:09.965855 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.944822    4421 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 20:01:09.965855 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.945814    4421 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 20:01:09.965855 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.945860    4421 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 20:01:09.965855 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.946033    4421 server.go:1014] "Starting client certificate rotation"
Nov 03 20:01:09.965855 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.946177    4421 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 20:01:09.965855 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.946232    4421 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:55:29 +0000 UTC" deadline="2028-07-23 03:41:18.422866856 +0000 UTC"
Nov 03 20:01:09.965855 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.946273    4421 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15055h40m8.476595891s"
Nov 03 20:01:09.965855 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.946505    4421 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:01:09.965855 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.947582    4421 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 20:01:09.966548 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.948986    4421 log.go:25] "Validated CRI v1 runtime API"
Nov 03 20:01:09.966548 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.950487    4421 log.go:25] "Validated CRI v1 image API"
Nov 03 20:01:09.966548 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.951140    4421 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 20:01:09.966548 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.953140    4421 fs.go:135] Filesystem UUIDs: map[318f595b-ab71-426b-90bf-36ab2c4d05d7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e36ff02b-8f16-4764-8530-928f047586f2:/dev/nvme0n1p4]
Nov 03 20:01:09.966765 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.953172    4421 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/072b4d3337117c5ea0ce179d4b73bad04b5790a500582f1468d6595e2fed9a06/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/072b4d3337117c5ea0ce179d4b73bad04b5790a500582f1468d6595e2fed9a06/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1bd82a08adbfcc7c587ec320745a85f7621d600dc1ceb21ab2e4329239ab1781/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1bd82a08adbfcc7c587ec320745a85f7621d600dc1ceb21ab2e4329239ab1781/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1cc9c89c3e02a465870ec2e297a5829ce2d3201cd7547f8409e67fdf0eb3a34f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1cc9c89c3e02a465870ec2e297a5829ce2d3201cd7547f8409e67fdf0eb3a34f/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4239b26cc8f0a599ac22e14a1a895b37df562b959267d15f1d470e8d0d12c484/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4239b26cc8f0a599ac22e14a1a895b37df562b959267d15f1d470e8d0d12c484/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4812e27e425a4a2b5c830a46560c0f9fee370441a6f57cd7a1c1eb5ab2ab54fa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4812e27e425a4a2b5c830a46560c0f9fee370441a6f57cd7a1c1eb5ab2ab54fa/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5354369e587df64c92b52a4ef40995576fc57bab04f5b23aa525b810cc8751e9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5354369e587df64c92b52a4ef40995576fc57bab04f5b23aa525b810cc8751e9/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/66ee5f1c87b8eb08c4370e0e3faef18cef3bf7921d8a748e5845e197c1337fb7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/66ee5f1c87b8eb08c4370e0e3faef18cef3bf7921d8a748e5845e197c1337fb7/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d5b9e407b152f10b72d3b5b458318d2f902da2c03f303f1476c8f0f66f07578/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d5b9e407b152f10b72d3b5b458318d2f902da2c03f303f1476c8f0f66f07578/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7651c81fe1cd14150be05fdd1805ebdf3147180082460cc1554f5bcd9780afb7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7651c81fe1cd14150be05fdd1805ebdf3147180082460cc1554f5bcd9780afb7/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ec457fb89d4030aa2e44e14bf257664354820bff93167c2813636bc76df33cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ec457fb89d4030aa2e44e14bf257664354820bff93167c2813636bc76df33cb/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a24ce8f1bb29443403798955cc34c0a42dee8258d23bda54a0757d54f49edcb1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a24ce8f1bb29443403798955cc34c0a42dee8258d23bda54a0757d54f49edcb1/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c6bb9956239dd70aaa24e2daa761f3809e82a5c3d2ac772b236ae77199e8d584/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c6bb9956239dd70aaa24e2daa761f3809e82a5c3d2ac772b236ae77199e8d584/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f04e18246bd8e09ce84e96e49436083094cdbe6c95ec881d4748a5645dd718dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f04e18246bd8e09ce84e96e49436083094cdbe6c95ec881d4748a5645dd718dc/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f47a37dac525b92bc09e460b56f9cd458f1074daf1136c74e3f6247f5b025d21/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f47a37dac525b92bc09e460b56f9cd458f1074daf1136c74e3f6247f5b025d21/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0181bad8-3459-4c64-8e31-12395fd4c6c7/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/0181bad8-3459-4c64-8e31-12395fd4c6c7/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07e7cf29-701a-4c9d-a3ad-12d4846157c2/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/07e7cf29-701a-4c9d-a3ad-12d4846157c2/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/126ae513-97e3-4ef0-b415-5ec03ea09960/volumes/kubernetes.io~projected/kube-api-access-n79bw:{mountpoint:/var/lib/kubelet/pods/126ae513-97e3-4ef0-b415-5ec03ea09960/volumes/kubernetes.io~projected/kube-api-access-n79bw major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/126ae513-97e3-4ef0-b415-5ec03ea09960/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/126ae513-97e3-4ef0-b415-5ec03ea09960/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/153d1ce3-22c2-444c-9bd2-ee91ff044d29/volumes/kubernetes.io~projected/kube-api-access-g525b:{mountpoint:/var/lib/kubelet/pods/153d1ce3-22c2-444c-9bd2-ee91ff044d29/volumes/kubernetes.io~projected/kube-api-access-g525b major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435/volumes/kubernetes.io~projected/kube-api-access-552wl:{mountpoint:/var/lib/kubelet/pods/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435/volumes/kubernetes.io~projected/kube-api-access-552wl major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3492be51-5e92-4369-8fc5-85d5ce0523fd/volumes/kubernetes.io~projected/kube-api-access-5k9l5:{mountpoint:/var/lib/kubelet/pods/3492be51-5e92-4369-8fc5-85d5ce0523fd/volumes/kubernetes.io~projected/kube-api-access-5k9l5 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/368fd902-a222-4a5b-9bbf-a8decb45ad31/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/368fd902-a222-4a5b-9bbf-a8decb45ad31/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/368fd902-a222-4a5b-9bbf-a8decb45ad31/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/368fd902-a222-4a5b-9bbf-a8decb45ad31/volumes/kubernetes.io~empty-dir/tmp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/368fd902-a222-4a5b-9bbf-a8decb45ad31/volumes/kubernetes.io~projected/kube-api-access-7b8s7:{mountpoint:/var/lib/kubelet/pods/368fd902-a222-4a5b-9bbf-a8decb45ad31/volumes/kubernetes.io~projected/kube-api-access-7b8s7 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/626900f3-1832-46a5-8c82-70d09284614c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/626900f3-1832-46a5-8c82-70d09284614c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/626900f3-1832-46a5-8c82-70d09284614c/volumes/kubernetes.io~projected/kube-api-access-pkrkh:{mountpoint:/var/lib/kubelet/pods/626900f3-1832-46a5-8c82-70d09284614c/volumes/kubernetes.io~projected/kube-api-access-pkrkh major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/626900f3-1832-46a5-8c82-70d09284614c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/626900f3-1832-46a5-8c82-70d09284614c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e8b3941-ab89-4838-bf55-6e9521ed259c/volumes/kubernetes.io~projected/kube-api-access-dpxxg:{mountpoint:/var/lib/kubelet/pods/6e8b3941-ab89-4838-bf55-6e9521ed259c/volumes/kubernetes.io~projected/kube-api-access-dpxxg major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71697994-0833-4570-83d3-afa5739725e9/volumes/kubernetes.io~projected/kube-api-access-rrkkm:{mountpoint:/var/lib/kubelet/pods/71697994-0833-4570-83d3-afa5739725e9/volumes/kubernetes.io~projected/kube-api-access-rrkkm major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78003043-36f1-452a-8ccf-37b574877715/volumes/kubernetes.io~projected/kube-api-access-gv682:{mountpoint:/var/lib/kubelet/pods/78003043-36f1-452a-8ccf-37b574877715/volumes/kubernetes.io~projected/kube-api-access-gv682 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78748ccd-0718-470e-ade1-74ff636db6b7/volumes/kubernetes.io~projected/kube-api-access-gtcnf:{mountpoint:/var/lib/kubelet/pods/78748ccd-0718-470e-ade1-74ff636db6b7/volumes/kubernetes.io~projected/kube-api-access-gtcnf major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78748ccd-0718-470e-ade1-74ff636db6b7/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/78748ccd-0718-470e-ade1-74ff636db6b7/volumes/kubernetes.io~secret/signing-key major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d61c96a-2363-47b2-949a-300d9b084779/volumes/kubernetes.io~projected/kube-api-access-j5q9m:{mountpoint:/var/lib/kubelet/pods/9d61c96a-2363-47b2-949a-300d9b084779/volumes/kubernetes.io~projected/kube-api-access-j5q9m major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac4c4fc1-a4c5-4f00-97bc-b201eede1226/volumes/kubernetes.io~projected/kube-api-access-tjlck:{mountpoint:/var/lib/kubelet/pods/ac4c4fc1-a4c5-4f00-97bc-b201eede1226/volumes/kubernetes.io~projected/kube-api-access-tjlck major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c5965c2f-be74-431c-a8ff-86598070072a/volumes/kubernetes.io~projected/kube-api-access-6mlgw:{mountpoint:/var/lib/kubelet/pods/c5965c2f-be74-431c-a8ff-86598070072a/volumes/kubernetes.io~projected/kube-api-access-6mlgw major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eaaa35b2-aee2-40f8-956b-0b9f697dc994/volumes/kubernetes.io~projected/kube-api-access-qzfmw:{mountpoint:/var/lib/kubelet/pods/eaaa35b2-aee2-40f8-956b-0b9f697dc994/volumes/kubernetes.io~projected/kube-api-access-qzfmw major:0 minor:202 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c1b93f3915cf83f1eef6fd4f4e964f96640bb93ffe684dbf3f857048a97295b8/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/4d93bf10f65382ac5e7e55102b281ed53384fdddbfa350b1a27f2fb7d4b139e9/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/1dd158feabee40a77b81b32748c89f667b32c3eaac4f44e16994d37fc25eb9de/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/a9dba641945cb5b38d5986c3a88140ef30c2841ad37444c311969dd6900e7dcd/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/2032554125befc7d54ca2f5bac681158d33422692371efe92ff9f0be71d4e85d/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/a678797506613bd8d38ee7de0d4791b98dc7ad8072c75284bdd0ea8714e51216/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-121:{mountpoint:/var/lib/containers/storage/overlay/4aaff08adf8e1e4f190ba6286fef91439069a26e3425c32ab6b8cda86a0ab709/merged major:0 minor:121 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e87c2414862d09a49974f6818af04a660fa4ed6be5098a240d8372baad109e28/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-128:{mountpoint:/var/lib/containers/storage/overlay/caa7953b06911f7ebfbaa9b70292916fcad09d7f471ca44bde56464de7338aee/merged major:0 minor:128 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/9491af5542dd4d9abda42d2b4316cccb708228028128f0512b03eab7a4e7b0c1/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/0b83c57f01e63820f6322451a1f23323b0584b6612f3ef50e724be478ef98216/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/57a50ed637c302bce59ad8418dc5f0dcdc363b5895e0c1cb9a8f4fd6b57e5cc7/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/c3a61948633ba056934a8bf2a551b9c98ce643d8e99b74f43d16cd5c89c8316c/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/dd0c4c0e453b94d8192b7754378a1d928c84c1ef585bcee3549101d3d7dae63a/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/b128849f4fdc7b7c834709fc9f7eb8e1d666d580195c7179ad516ee87a627496/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/d0029a6523db8adfe9b1f7ac52ee1b1bf7013efc14b3919dc52bfab1bd53d1e1/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/29653792e47c42e3f1891a7bd97e7977c6c2f80106d52e66c05b18c77a0b6eb6/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d89b508155260ed72ceca9d3fb43aa38b05b93c701226beab0199c43b7e73177/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/ff24efdbd8b12191ff4b3c92152c4722ce65981b35088db1d327d30836595ca6/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/bb480662738c5e04a46ccc48b3d5be56528ba3754423dc98f3ffa8228c806cae/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/f51371b5e323c8a07e5e06f8e87dd63dce42c4ef3448f67c647edb461c93f418/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/f0f63e8d6d066bf60a2080af15f221b5db308a949d18cbc9d8187bf23b1cf576/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/db18d7a3449e1d76745b2fd2eac06239a3868af9c7293fe9716c5b360a7737a7/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/529b7cf4ed09f5aa4d22bc61d829c11c679dd4fd9c7fd2443f2bde3b1a5507f4/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/9647c81ff16900b118dde4dd8dafbca0c2ee561b78e5da5ddff1387fcad890db/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/8e06c0c1afa8801a5e25380e0919ae1a716332df5577c6d5ea3631e764b2fbff/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/73ad051299156fa819093c1a17e294dec0febc336211d21d781dcf6b702e0f83/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/afa5c222b32e77711a3a78f68c4959b734cb72e6e81eb2cbbd2dfe1063c44478/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/0d35aee29e66a40cbf95751a00b2f485c87a7f1889f89701a9fde9bdf328d273/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/68d79ab3c6f641dff4dadb590a09a2dedec806c10305c71862c4652408a03b1d/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/cbd9698b80ede4246364c4dc34fcd7ac229813327035552f0c284381d7006c96/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/05f1eeaf3b14d53105e8663aedd1d908028dcaea364cb6d578adac355272ddb0/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/8754c489b323dab13412f31e16f6a24845c7cd6998564760ad45c37ee1d0e23a/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/9ff6240aff90851dec6df2c7d7445336119b3d28ba252be5cb50cd04132954d3/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/e29b5448f7240ac409486f6dc6d6d52b4715a488557a23aae384a802f0cb483c/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/c72f591d2c910187731fc4fdfe23e8d70eb4d70886ba88ee9f4f9faeb0a42270/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/711684f59e489d564d315581af185350b4c8b44870077262c2d73dc0fb3cd767/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/97dc4ccdacacc7d60af9eadcc5b7484a6c6b651229d2d07744d7b74431125b98/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/e118fe6f33d1c403638e6dc16d9aef1beacdda610fea7d24923130f837b6138d/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 20:01:09.966988 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.962046    4421 manager.go:217] Machine: {Timestamp:2026-11-03 20:01:09.961571938 +0000 UTC m=+0.094700393 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3122926 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b5f5b51f5c786985616f94002379c SystemUUID:ec2b5f5b-51f5-c786-9856-16f94002379c BootID:fe805aa2-b88f-417f-8a4c-154126457afa Filesystems:[{Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/626900f3-1832-46a5-8c82-70d09284614c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4239b26cc8f0a599ac22e14a1a895b37df562b959267d15f1d470e8d0d12c484/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c6bb9956239dd70aaa24e2daa761f3809e82a5c3d2ac772b236ae77199e8d584/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78748ccd-0718-470e-ade1-74ff636db6b7/volumes/kubernetes.io~projected/kube-api-access-gtcnf DeviceMajor:0 DeviceMinor:214 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3492be51-5e92-4369-8fc5-85d5ce0523fd/volumes/kubernetes.io~projected/kube-api-access-5k9l5 DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1cc9c89c3e02a465870ec2e297a5829ce2d3201cd7547f8409e67fdf0eb3a34f/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4812e27e425a4a2b5c830a46560c0f9fee370441a6f57cd7a1c1eb5ab2ab54fa/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ec457fb89d4030aa2e44e14bf257664354820bff93167c2813636bc76df33cb/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c5965c2f-be74-431c-a8ff-86598070072a/volumes/kubernetes.io~projected/kube-api-access-6mlgw DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5354369e587df64c92b52a4ef40995576fc57bab04f5b23aa525b810cc8751e9/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78748ccd-0718-470e-ade1-74ff636db6b7/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ac4c4fc1-a4c5-4f00-97bc-b201eede1226/volumes/kubernetes.io~projected/kube-api-access-tjlck DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/626900f3-1832-46a5-8c82-70d09284614c/volumes/kubernetes.io~projected/kube-api-access-pkrkh DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f47a37dac525b92bc09e460b56f9cd458f1074daf1136c74e3f6247f5b025d21/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e8b3941-ab89-4838-bf55-6e9521ed259c/volumes/kubernetes.io~projected/kube-api-access-dpxxg DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/368fd902-a222-4a5b-9bbf-a8decb45ad31/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a24ce8f1bb29443403798955cc34c0a42dee8258d23bda54a0757d54f49edcb1/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/66ee5f1c87b8eb08c4370e0e3faef18cef3bf7921d8a748e5845e197c1337fb7/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/072b4d3337117c5ea0ce179d4b73bad04b5790a500582f1468d6595e2fed9a06/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1bd82a08adbfcc7c587ec320745a85f7621d600dc1ceb21ab2e4329239ab1781/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7651c81fe1cd14150be05fdd1805ebdf3147180082460cc1554f5bcd9780afb7/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/368fd902-a222-4a5b-9bbf-a8decb45ad31/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435/volumes/kubernetes.io~projected/kube-api-access-552wl DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/07e7cf29-701a-4c9d-a3ad-12d4846157c2/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/126ae513-97e3-4ef0-b415-5ec03ea09960/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/126ae513-97e3-4ef0-b415-5ec03ea09960/volumes/kubernetes.io~projected/kube-api-access-n79bw DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0181bad8-3459-4c64-8e31-12395fd4c6c7/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/626900f3-1832-46a5-8c82-70d09284614c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/153d1ce3-22c2-444c-9bd2-ee91ff044d29/volumes/kubernetes.io~projected/kube-api-access-g525b DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d5b9e407b152f10b72d3b5b458318d2f902da2c03f303f1476c8f0f66f07578/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-121 DeviceMajor:0 DeviceMinor:121 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-128 DeviceMajor:0 DeviceMinor:128 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d61c96a-2363-47b2-949a-300d9b084779/volumes/kubernetes.io~projected/kube-api-access-j5q9m DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/78003043-36f1-452a-8ccf-37b574877715/volumes/kubernetes.io~projected/kube-api-access-gv682 DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f04e18246bd8e09ce84e96e49436083094cdbe6c95ec881d4748a5645dd718dc/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/368fd902-a222-4a5b-9bbf-a8decb45ad31/volumes/kubernetes.io~projected/kube-api-access-7b8s7 DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eaaa35b2-aee2-40f8-956b-0b9f697dc994/volumes/kubernetes.io~projected/kube-api-access-qzfmw DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/71697994-0833-4570-83d3-afa5739725e9/volumes/kubernetes.io~projected/kube-api-access-rrkkm DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:4239b26cc8f0a59 MacAddress:d6:24:61:2d:33:9a Speed:10000 Mtu:8901} {Name:5354369e587df64 MacAddress:a6:53:73:95:8f:97 Speed:10000 Mtu:8901} {Name:7651c81fe1cd141 MacAddress:b2:57:a6:69:25:15 Speed:10000 Mtu:8901} {Name:8ec457fb89d4030 MacAddress:26:3a:6a:6b:05:d9 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e5:b6:35:8d Speed:0 Mtu:9001} {Name:br-int MacAddress:7e:a1:d9:5a:84:b6 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:e5:b6:35:8d Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:a6:51:af:36:fd:f0 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:16:59:ad:98:cb:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 20:01:09.966988 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.962289    4421 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 20:01:09.966988 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.962329    4421 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 20:01:09.966988 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.962586    4421 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 20:01:09.966988 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.962620    4421 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-48.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 20:01:09.966988 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.962743    4421 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 20:01:09.966988 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.962750    4421 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 20:01:09.966988 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.962764    4421 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:01:09.966988 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.962776    4421 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 20:01:09.966988 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.962849    4421 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 20:01:09.966988 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.962997    4421 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 20:01:09.966988 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.963092    4421 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 20:01:09.966988 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.963124    4421 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 20:01:09.966988 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.963136    4421 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 20:01:09.966988 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.963145    4421 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 20:01:09.966988 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.963153    4421 kubelet.go:405] "Adding apiserver pod source"
Nov 03 20:01:09.966988 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.963161    4421 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 20:01:09.966988 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.963696    4421 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 20:01:09.966988 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.963705    4421 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:01:09.966988 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.964218    4421 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 20:01:09.966988 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.966920    4421 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 20:01:09.966988 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.966933    4421 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 20:01:09.968149 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.967056    4421 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 20:01:09.968149 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.967305    4421 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 20:01:09.968149 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.967350    4421 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 20:01:09.968149 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.967461    4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 20:01:09.968149 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.967479    4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 20:01:09.968149 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.967488    4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 20:01:09.968149 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.967497    4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 20:01:09.968149 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.967505    4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 20:01:09.968149 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.967515    4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 20:01:09.968149 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.967524    4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 20:01:09.968149 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.967533    4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 20:01:09.968149 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.967545    4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 20:01:09.968149 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.967554    4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 20:01:09.968149 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.967574    4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 20:01:09.968149 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.967586    4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 20:01:09.968149 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.967641    4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 20:01:09.968149 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.967652    4421 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 20:01:09.971234 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.971218    4421 server.go:1263] "Started kubelet"
Nov 03 20:01:09.971397 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.971328    4421 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 20:01:09.971445 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.971432    4421 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 20:01:09.971722 ip-10-0-143-48 systemd[1]: Started Kubernetes Kubelet.
Nov 03 20:01:09.974233 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.972467    4421 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 20:01:09.974233 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.972920    4421 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 20:01:09.974998 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.974979    4421 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 20:01:09.975470 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.975454    4421 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.976158    4421 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.976432    4421 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:55:31 +0000 UTC" deadline="2028-08-06 10:28:22.368146202 +0000 UTC"
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.976469    4421 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15398h27m12.391680081s"
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:01:09.976508    4421 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-48.us-east-2.compute.internal\" not found"
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.976542    4421 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.976551    4421 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.976989    4421 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979479    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78748ccd-0718-470e-ade1-74ff636db6b7" volumeName="kubernetes.io/configmap/78748ccd-0718-470e-ade1-74ff636db6b7-signing-cabundle" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979511    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d61c96a-2363-47b2-949a-300d9b084779" volumeName="kubernetes.io/configmap/9d61c96a-2363-47b2-949a-300d9b084779-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979543    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07e7cf29-701a-4c9d-a3ad-12d4846157c2" volumeName="kubernetes.io/configmap/07e7cf29-701a-4c9d-a3ad-12d4846157c2-konnectivity-ca" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979562    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="626900f3-1832-46a5-8c82-70d09284614c" volumeName="kubernetes.io/configmap/626900f3-1832-46a5-8c82-70d09284614c-env-overrides" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979575    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71697994-0833-4570-83d3-afa5739725e9" volumeName="kubernetes.io/projected/71697994-0833-4570-83d3-afa5739725e9-kube-api-access-rrkkm" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979591    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eaaa35b2-aee2-40f8-956b-0b9f697dc994" volumeName="kubernetes.io/configmap/eaaa35b2-aee2-40f8-956b-0b9f697dc994-config-volume" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979630    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="368fd902-a222-4a5b-9bbf-a8decb45ad31" volumeName="kubernetes.io/empty-dir/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-tuned" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979646    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="153d1ce3-22c2-444c-9bd2-ee91ff044d29" volumeName="kubernetes.io/empty-dir/153d1ce3-22c2-444c-9bd2-ee91ff044d29-tmp-dir" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979661    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="626900f3-1832-46a5-8c82-70d09284614c" volumeName="kubernetes.io/configmap/626900f3-1832-46a5-8c82-70d09284614c-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979692    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="626900f3-1832-46a5-8c82-70d09284614c" volumeName="kubernetes.io/projected/626900f3-1832-46a5-8c82-70d09284614c-kube-api-access-pkrkh" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979702    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="626900f3-1832-46a5-8c82-70d09284614c" volumeName="kubernetes.io/secret/626900f3-1832-46a5-8c82-70d09284614c-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979715    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e8b3941-ab89-4838-bf55-6e9521ed259c" volumeName="kubernetes.io/projected/6e8b3941-ab89-4838-bf55-6e9521ed259c-kube-api-access-dpxxg" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979731    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78003043-36f1-452a-8ccf-37b574877715" volumeName="kubernetes.io/projected/78003043-36f1-452a-8ccf-37b574877715-kube-api-access-gv682" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979744    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78748ccd-0718-470e-ade1-74ff636db6b7" volumeName="kubernetes.io/secret/78748ccd-0718-470e-ade1-74ff636db6b7-signing-key" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979763    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3492be51-5e92-4369-8fc5-85d5ce0523fd" volumeName="kubernetes.io/projected/3492be51-5e92-4369-8fc5-85d5ce0523fd-kube-api-access-5k9l5" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979776    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78748ccd-0718-470e-ade1-74ff636db6b7" volumeName="kubernetes.io/projected/78748ccd-0718-470e-ade1-74ff636db6b7-kube-api-access-gtcnf" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979791    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d61c96a-2363-47b2-949a-300d9b084779" volumeName="kubernetes.io/configmap/9d61c96a-2363-47b2-949a-300d9b084779-cni-binary-copy" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979802    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac4c4fc1-a4c5-4f00-97bc-b201eede1226" volumeName="kubernetes.io/configmap/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-multus-daemon-config" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979815    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac4c4fc1-a4c5-4f00-97bc-b201eede1226" volumeName="kubernetes.io/projected/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-kube-api-access-tjlck" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979863    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5965c2f-be74-431c-a8ff-86598070072a" volumeName="kubernetes.io/projected/c5965c2f-be74-431c-a8ff-86598070072a-kube-api-access-6mlgw" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979875    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="368fd902-a222-4a5b-9bbf-a8decb45ad31" volumeName="kubernetes.io/projected/368fd902-a222-4a5b-9bbf-a8decb45ad31-kube-api-access-7b8s7" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979892    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20b2d254-c5b2-4a59-8ce9-49a9f2c0f435" volumeName="kubernetes.io/empty-dir/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-data-volume" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979903    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="368fd902-a222-4a5b-9bbf-a8decb45ad31" volumeName="kubernetes.io/empty-dir/368fd902-a222-4a5b-9bbf-a8decb45ad31-tmp" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979915    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78003043-36f1-452a-8ccf-37b574877715" volumeName="kubernetes.io/configmap/78003043-36f1-452a-8ccf-37b574877715-serviceca" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979932    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d61c96a-2363-47b2-949a-300d9b084779" volumeName="kubernetes.io/projected/9d61c96a-2363-47b2-949a-300d9b084779-kube-api-access-j5q9m" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979945    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac4c4fc1-a4c5-4f00-97bc-b201eede1226" volumeName="kubernetes.io/configmap/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-cni-binary-copy" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979962    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c5965c2f-be74-431c-a8ff-86598070072a" volumeName="kubernetes.io/configmap/c5965c2f-be74-431c-a8ff-86598070072a-iptables-alerter-script" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.979979    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="126ae513-97e3-4ef0-b415-5ec03ea09960" volumeName="kubernetes.io/projected/126ae513-97e3-4ef0-b415-5ec03ea09960-kube-api-access-n79bw" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.980005    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="126ae513-97e3-4ef0-b415-5ec03ea09960" volumeName="kubernetes.io/secret/126ae513-97e3-4ef0-b415-5ec03ea09960-cert" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.980016    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20b2d254-c5b2-4a59-8ce9-49a9f2c0f435" volumeName="kubernetes.io/configmap/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.980032    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d61c96a-2363-47b2-949a-300d9b084779" volumeName="kubernetes.io/configmap/9d61c96a-2363-47b2-949a-300d9b084779-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.980038    4421 factory.go:55] Registering systemd factory
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.980043    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eaaa35b2-aee2-40f8-956b-0b9f697dc994" volumeName="kubernetes.io/empty-dir/eaaa35b2-aee2-40f8-956b-0b9f697dc994-tmp-dir" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.980052    4421 factory.go:223] Registration of the systemd container factory successfully
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.980055    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0181bad8-3459-4c64-8e31-12395fd4c6c7" volumeName="kubernetes.io/secret/0181bad8-3459-4c64-8e31-12395fd4c6c7-original-pull-secret" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.980071    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eaaa35b2-aee2-40f8-956b-0b9f697dc994" volumeName="kubernetes.io/projected/eaaa35b2-aee2-40f8-956b-0b9f697dc994-kube-api-access-qzfmw" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.980084    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07e7cf29-701a-4c9d-a3ad-12d4846157c2" volumeName="kubernetes.io/secret/07e7cf29-701a-4c9d-a3ad-12d4846157c2-agent-certs" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.980101    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="153d1ce3-22c2-444c-9bd2-ee91ff044d29" volumeName="kubernetes.io/projected/153d1ce3-22c2-444c-9bd2-ee91ff044d29-kube-api-access-g525b" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.980114    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20b2d254-c5b2-4a59-8ce9-49a9f2c0f435" volumeName="kubernetes.io/projected/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-kube-api-access-552wl" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.980129    4421 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="626900f3-1832-46a5-8c82-70d09284614c" volumeName="kubernetes.io/configmap/626900f3-1832-46a5-8c82-70d09284614c-ovnkube-config" seLinuxMountContext=""
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.980144    4421 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.980150    4421 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.980892    4421 factory.go:153] Registering CRI-O factory
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.980910    4421 factory.go:223] Registration of the crio container factory successfully
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.980966    4421 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 20:01:09.980957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.980992    4421 factory.go:103] Registering Raw factory
Nov 03 20:01:09.983230 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.981014    4421 manager.go:1199] Started watching for new ooms in manager
Nov 03 20:01:09.983230 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.982036    4421 manager.go:319] Starting recovery of all containers
Nov 03 20:01:09.983230 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.982115    4421 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:01:09.985763 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.984577    4421 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:01:09.985763 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.984930    4421 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:01:09.985763 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:09.985053    4421 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 20:01:10.005932 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.005891    4421 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 20:01:10.007566 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.007543    4421 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 20:01:10.007665 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.007569    4421 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 20:01:10.007665 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.007613    4421 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 20:01:10.007665 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:01:10.007654    4421 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 20:01:10.011813 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.011624    4421 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 20:01:10.033575 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.033494    4421 generic.go:356] "Generic (PLEG): container finished" podID="9d61c96a-2363-47b2-949a-300d9b084779" containerID="715874c6b232170f5ec7d0d3ffbd1e914d9c8e6184cdc8f8961687c325d498c9" exitCode=0
Nov 03 20:01:10.033575 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.033543    4421 generic.go:356] "Generic (PLEG): container finished" podID="9d61c96a-2363-47b2-949a-300d9b084779" containerID="891162d75aebe49b104c7a6361b195048a38f8e912012c6e97df4f9800ce0377" exitCode=0
Nov 03 20:01:10.033575 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.033553    4421 generic.go:356] "Generic (PLEG): container finished" podID="9d61c96a-2363-47b2-949a-300d9b084779" containerID="3e6b28af71e66bdab796ea5999c446c83ca51b1361a583f82e8c53ab8abc64d1" exitCode=0
Nov 03 20:01:10.033575 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.033563    4421 generic.go:356] "Generic (PLEG): container finished" podID="9d61c96a-2363-47b2-949a-300d9b084779" containerID="ed24c50e209a0962486fd8774e1154afef6119dfc7499c20853b0e8e3a8f1e38" exitCode=0
Nov 03 20:01:10.033575 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.033573    4421 generic.go:356] "Generic (PLEG): container finished" podID="9d61c96a-2363-47b2-949a-300d9b084779" containerID="c37be76c398d4cbf1a38bc3024327db2064b247dc54f2fcfdb08d1476fa95252" exitCode=0
Nov 03 20:01:10.033856 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.033583    4421 generic.go:356] "Generic (PLEG): container finished" podID="9d61c96a-2363-47b2-949a-300d9b084779" containerID="1bd35de346bfe6419bd34f96a66c09cb4b4cdece3bc45b90195becd4c71fc8cb" exitCode=0
Nov 03 20:01:10.035563 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.035535    4421 generic.go:356] "Generic (PLEG): container finished" podID="b45f8e0dedc76d41e693c990769490b8" containerID="9b5b24f3311434dd42b4ae883fca519bad8b0d0c1ab3dd8330cbc8471a674212" exitCode=0
Nov 03 20:01:10.049485 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.049464    4421 manager.go:324] Recovery completed
Nov 03 20:01:10.063205 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.063182    4421 cpu_manager.go:226] "Starting" policy="none"
Nov 03 20:01:10.063205 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.063201    4421 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 20:01:10.063346 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.063220    4421 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 20:01:10.063376 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.063361    4421 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 20:01:10.063404 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.063376    4421 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 20:01:10.063404 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.063400    4421 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 20:01:10.063510 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.063408    4421 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 20:01:10.063510 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.063415    4421 policy_none.go:50] "Start"
Nov 03 20:01:10.063510 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.063428    4421 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 20:01:10.063510 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.063442    4421 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 20:01:10.063631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.063573    4421 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 20:01:10.063631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.063580    4421 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 20:01:10.063631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.063586    4421 policy_none.go:44] "Start"
Nov 03 20:01:10.069135 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.069106    4421 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 20:01:10.072613 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.072588    4421 manager.go:341] "Starting Device Plugin manager"
Nov 03 20:01:10.073145 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:01:10.072623    4421 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 20:01:10.073224 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.073148    4421 server.go:87] "Starting device plugin registration server"
Nov 03 20:01:10.073393 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.073381    4421 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 20:01:10.073454 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.073396    4421 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 20:01:10.073520 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.073500    4421 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 20:01:10.073659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.073625    4421 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 20:01:10.073727 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.073668    4421 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 20:01:10.073727 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.073675    4421 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 20:01:10.073838 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.073728    4421 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:01:10.073657342Z","UUID":"bebe03b1-9d09-41be-af0d-db2f2927970f","Handler":null,"Name":"","Endpoint":""}
Nov 03 20:01:10.074792 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:01:10.074769    4421 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 20:01:10.074938 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.074907    4421 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 20:01:10.075018 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.074944    4421 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 20:01:10.108444 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.108399    4421 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal"]
Nov 03 20:01:10.108931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.108888    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal" event={"ID":"b45f8e0dedc76d41e693c990769490b8","Type":"ContainerStarted","Data":"140ab387404d67c43d205b26dd6f41bc2961ce4070715fb41eefd71e9362f005"}
Nov 03 20:01:10.108931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.108932    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal" event={"ID":"b45f8e0dedc76d41e693c990769490b8","Type":"ContainerDied","Data":"9b5b24f3311434dd42b4ae883fca519bad8b0d0c1ab3dd8330cbc8471a674212"}
Nov 03 20:01:10.109113 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.108949    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal" event={"ID":"b45f8e0dedc76d41e693c990769490b8","Type":"ContainerStarted","Data":"f47a37dac525b92bc09e460b56f9cd458f1074daf1136c74e3f6247f5b025d21"}
Nov 03 20:01:10.109113 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.108974    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal" event={"ID":"cc3d49b24d21c77c008662d39f9bb541","Type":"ContainerStarted","Data":"822a9cc5a83b8416ab77a8cc625fbb7769bd3eff45c26476a364f428d933fba1"}
Nov 03 20:01:10.109113 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.108978    4421 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:01:10.109113 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.108992    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal" event={"ID":"cc3d49b24d21c77c008662d39f9bb541","Type":"ContainerStarted","Data":"072b4d3337117c5ea0ce179d4b73bad04b5790a500582f1468d6595e2fed9a06"}
Nov 03 20:01:10.109113 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.109095    4421 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:01:10.123072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.123054    4421 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:01:10.123139 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:01:10.123096    4421 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:01:10.123648 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.123634    4421 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:01:10.123687 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:01:10.123671    4421 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:01:10.173787 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.173765    4421 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 20:01:10.175067 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.175049    4421 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-48.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 20:01:10.175176 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.175087    4421 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-48.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 20:01:10.175444 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.175421    4421 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-48.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 20:01:10.175524 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.175511    4421 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:01:10.185292 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.185267    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b45f8e0dedc76d41e693c990769490b8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal\" (UID: \"b45f8e0dedc76d41e693c990769490b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:01:10.185391 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.185295    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc3d49b24d21c77c008662d39f9bb541-config\") pod \"kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal\" (UID: \"cc3d49b24d21c77c008662d39f9bb541\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:01:10.185391 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.185315    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b45f8e0dedc76d41e693c990769490b8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal\" (UID: \"b45f8e0dedc76d41e693c990769490b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:01:10.188932 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.188917    4421 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:01:10.188985 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.188975    4421 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:01:10.286245 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.286168    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b45f8e0dedc76d41e693c990769490b8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal\" (UID: \"b45f8e0dedc76d41e693c990769490b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:01:10.286245 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.286212    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b45f8e0dedc76d41e693c990769490b8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal\" (UID: \"b45f8e0dedc76d41e693c990769490b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:01:10.286398 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.286257    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc3d49b24d21c77c008662d39f9bb541-config\") pod \"kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal\" (UID: \"cc3d49b24d21c77c008662d39f9bb541\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:01:10.286398 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.286273    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b45f8e0dedc76d41e693c990769490b8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal\" (UID: \"b45f8e0dedc76d41e693c990769490b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:01:10.286398 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.286302    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cc3d49b24d21c77c008662d39f9bb541-config\") pod \"kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal\" (UID: \"cc3d49b24d21c77c008662d39f9bb541\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:01:10.286398 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.286331    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b45f8e0dedc76d41e693c990769490b8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal\" (UID: \"b45f8e0dedc76d41e693c990769490b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:01:10.963807 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.963778    4421 apiserver.go:52] "Watching apiserver"
Nov 03 20:01:10.974048 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.974025    4421 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 20:01:10.974428 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.974410    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-h6crz","openshift-network-operator/iptables-alerter-v9qll","openshift-cluster-node-tuning-operator/tuned-w72hd","openshift-ingress-canary/ingress-canary-sx9rt","openshift-service-ca/service-ca-5fffc97bdf-brm42","openshift-dns/node-resolver-g2rp5","openshift-image-registry/node-ca-tt4df","openshift-multus/multus-additional-cni-plugins-86vz2","openshift-dns/dns-default-wb9sn","openshift-insights/insights-runtime-extractor-74j6n","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal","openshift-multus/multus-mq5xz","openshift-multus/network-metrics-daemon-286gh","kube-system/konnectivity-agent-k5b5s","openshift-ovn-kubernetes/ovnkube-node-lcp7r","kube-system/global-pull-secret-syncer-x8wrn","kube-system/kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal"]
Nov 03 20:01:10.975993 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.975973    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:01:10.978487 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.978465    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 20:01:10.979000 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.978978    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 20:01:10.979784 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.979760    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 20:01:10.979878 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.979763    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:10.979878 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.979805    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 20:01:10.980847 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.980830    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:10.984448 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.984412    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:01:10.984533 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.984505    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 20:01:10.984653 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.984633    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 20:01:10.984711 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.984691    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 20:01:10.984818 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.984803    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 20:01:10.985022 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.985003    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 20:01:10.985075 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.985018    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8fc4f\""
Nov 03 20:01:10.985180 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.985163    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 20:01:10.985393 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.985374    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 20:01:10.985461 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.985394    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gz7vs\""
Nov 03 20:01:10.985461 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.985406    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 20:01:10.985553 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.985470    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 20:01:10.985666 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.985646    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zc6h4\""
Nov 03 20:01:10.985722 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.985667    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vlwgf\""
Nov 03 20:01:10.985722 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.985710    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 20:01:10.986056 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.986037    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 20:01:10.986195 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.986178    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 20:01:10.986302 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.986286    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 20:01:10.986357 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.984812    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 20:01:10.986423 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.986040    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rnml9\""
Nov 03 20:01:10.986559 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.986536    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vbl6p\""
Nov 03 20:01:10.986792 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.986769    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 20:01:10.986871 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.986793    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nc2g5\""
Nov 03 20:01:10.986931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.986875    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 20:01:10.986931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.986877    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 20:01:10.986931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.986918    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 20:01:10.987053 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.986927    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lxhjg\""
Nov 03 20:01:10.987053 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.986971    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 20:01:10.987053 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.986976    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 20:01:10.987623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.987587    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s55dl\""
Nov 03 20:01:10.987698 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.987628    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zfqpt\""
Nov 03 20:01:10.987698 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.987635    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 20:01:10.987698 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.987645    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 20:01:10.987698 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.987677    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 20:01:10.987698 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.987683    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7vkxg\""
Nov 03 20:01:10.987901 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.987724    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 20:01:10.987901 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.987758    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 20:01:10.987901 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.987767    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 20:01:10.987901 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.987815    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 20:01:10.988027 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.987905    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 20:01:10.988027 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.987962    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 20:01:10.988115 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.988053    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kmt87\""
Nov 03 20:01:10.988163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.988142    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 20:01:10.988250 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.988227    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 20:01:10.988250 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.988235    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9mszh\""
Nov 03 20:01:10.988528 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.988512    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 20:01:10.988580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.988533    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-whl79\""
Nov 03 20:01:10.988650 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.988622    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-var-lib-kubelet\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:10.988650 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.988642    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-modprobe-d\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:10.988751 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.988661    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/78003043-36f1-452a-8ccf-37b574877715-serviceca\") pod \"node-ca-tt4df\" (UID: \"78003043-36f1-452a-8ccf-37b574877715\") " pod="openshift-image-registry/node-ca-tt4df"
Nov 03 20:01:10.988751 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.988686    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv682\" (UniqueName: \"kubernetes.io/projected/78003043-36f1-452a-8ccf-37b574877715-kube-api-access-gv682\") pod \"node-ca-tt4df\" (UID: \"78003043-36f1-452a-8ccf-37b574877715\") " pod="openshift-image-registry/node-ca-tt4df"
Nov 03 20:01:10.988843 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.988813    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5965c2f-be74-431c-a8ff-86598070072a-host-slash\") pod \"iptables-alerter-v9qll\" (UID: \"c5965c2f-be74-431c-a8ff-86598070072a\") " pod="openshift-network-operator/iptables-alerter-v9qll"
Nov 03 20:01:10.988843 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.988834    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-slash\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:10.988926 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.988853    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-cni-netd\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:10.988926 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.988899    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/626900f3-1832-46a5-8c82-70d09284614c-ovnkube-script-lib\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:10.988926 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.988921    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-var-lib-openvswitch\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:10.989035 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.988937    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-sysctl-d\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:10.989035 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.988951    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-var-lib-kubelet\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:10.989035 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.988965    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c5965c2f-be74-431c-a8ff-86598070072a-iptables-alerter-script\") pod \"iptables-alerter-v9qll\" (UID: \"c5965c2f-be74-431c-a8ff-86598070072a\") " pod="openshift-network-operator/iptables-alerter-v9qll"
Nov 03 20:01:10.989035 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.988989    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-run-openvswitch\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:10.989035 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989020    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-sys\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:10.989211 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989050    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/368fd902-a222-4a5b-9bbf-a8decb45ad31-tmp\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:10.989211 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989067    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/78003043-36f1-452a-8ccf-37b574877715-serviceca\") pod \"node-ca-tt4df\" (UID: \"78003043-36f1-452a-8ccf-37b574877715\") " pod="openshift-image-registry/node-ca-tt4df"
Nov 03 20:01:10.989211 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989085    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-kubelet\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:10.989211 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989110    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/368fd902-a222-4a5b-9bbf-a8decb45ad31-tmp\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:10.989211 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989152    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/626900f3-1832-46a5-8c82-70d09284614c-ovnkube-script-lib\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:10.989211 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989152    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:10.989211 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989183    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mlgw\" (UniqueName: \"kubernetes.io/projected/c5965c2f-be74-431c-a8ff-86598070072a-kube-api-access-6mlgw\") pod \"iptables-alerter-v9qll\" (UID: \"c5965c2f-be74-431c-a8ff-86598070072a\") " pod="openshift-network-operator/iptables-alerter-v9qll"
Nov 03 20:01:10.989491 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989240    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c5965c2f-be74-431c-a8ff-86598070072a-iptables-alerter-script\") pod \"iptables-alerter-v9qll\" (UID: \"c5965c2f-be74-431c-a8ff-86598070072a\") " pod="openshift-network-operator/iptables-alerter-v9qll"
Nov 03 20:01:10.989491 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989243    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-run-netns\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:10.989491 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989279    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-cnibin\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:10.989491 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989314    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-var-lib-cni-multus\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:10.989491 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989343    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-etc-kubernetes\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:10.989491 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989371    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjlck\" (UniqueName: \"kubernetes.io/projected/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-kube-api-access-tjlck\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:10.989491 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989394    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-sysconfig\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:10.989491 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989410    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0181bad8-3459-4c64-8e31-12395fd4c6c7-original-pull-secret\") pod \"global-pull-secret-syncer-x8wrn\" (UID: \"0181bad8-3459-4c64-8e31-12395fd4c6c7\") " pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:01:10.989491 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989483    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-run-ovn\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:10.989876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989502    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-multus-cni-dir\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:10.989876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989587    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-run-netns\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:10.989876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989645    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-multus-daemon-config\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:10.989876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989681    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0181bad8-3459-4c64-8e31-12395fd4c6c7-original-pull-secret\") pod \"global-pull-secret-syncer-x8wrn\" (UID: \"0181bad8-3459-4c64-8e31-12395fd4c6c7\") " pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:01:10.989876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989686    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0181bad8-3459-4c64-8e31-12395fd4c6c7-kubelet-config\") pod \"global-pull-secret-syncer-x8wrn\" (UID: \"0181bad8-3459-4c64-8e31-12395fd4c6c7\") " pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:01:10.989876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989710    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-log-socket\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:10.989876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989771    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-run\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:10.989876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989824    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/626900f3-1832-46a5-8c82-70d09284614c-ovnkube-config\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:10.989876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989851    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkrkh\" (UniqueName: \"kubernetes.io/projected/626900f3-1832-46a5-8c82-70d09284614c-kube-api-access-pkrkh\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:10.989876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989869    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-multus-daemon-config\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:10.990227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.989905    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-run-multus-certs\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:10.990227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990001    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-tuned\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:10.990227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990025    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/626900f3-1832-46a5-8c82-70d09284614c-ovnkube-config\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:10.990227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990030    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b8s7\" (UniqueName: \"kubernetes.io/projected/368fd902-a222-4a5b-9bbf-a8decb45ad31-kube-api-access-7b8s7\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:10.990227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990062    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-node-log\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:10.990227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990145    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-lrvk5\""
Nov 03 20:01:10.990227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990144    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-run-ovn-kubernetes\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:10.990227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990176    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-kubernetes\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:10.990227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990212    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-tuned\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:10.990227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990219    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/153d1ce3-22c2-444c-9bd2-ee91ff044d29-hosts-file\") pod \"node-resolver-g2rp5\" (UID: \"153d1ce3-22c2-444c-9bd2-ee91ff044d29\") " pod="openshift-dns/node-resolver-g2rp5"
Nov 03 20:01:10.990614 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990254    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-etc-openvswitch\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:10.990614 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990274    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-run-k8s-cni-cncf-io\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:10.990614 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990289    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3492be51-5e92-4369-8fc5-85d5ce0523fd-metrics-certs\") pod \"network-metrics-daemon-286gh\" (UID: \"3492be51-5e92-4369-8fc5-85d5ce0523fd\") " pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:01:10.990614 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990361    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/153d1ce3-22c2-444c-9bd2-ee91ff044d29-tmp-dir\") pod \"node-resolver-g2rp5\" (UID: \"153d1ce3-22c2-444c-9bd2-ee91ff044d29\") " pod="openshift-dns/node-resolver-g2rp5"
Nov 03 20:01:10.990614 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990390    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-run-systemd\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:10.990614 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990412    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-cni-binary-copy\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:10.990614 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990426    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-host\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:10.990614 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990481    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k9l5\" (UniqueName: \"kubernetes.io/projected/3492be51-5e92-4369-8fc5-85d5ce0523fd-kube-api-access-5k9l5\") pod \"network-metrics-daemon-286gh\" (UID: \"3492be51-5e92-4369-8fc5-85d5ce0523fd\") " pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:01:10.990614 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990524    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/153d1ce3-22c2-444c-9bd2-ee91ff044d29-tmp-dir\") pod \"node-resolver-g2rp5\" (UID: \"153d1ce3-22c2-444c-9bd2-ee91ff044d29\") " pod="openshift-dns/node-resolver-g2rp5"
Nov 03 20:01:10.990614 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990532    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxxg\" (UniqueName: \"kubernetes.io/projected/6e8b3941-ab89-4838-bf55-6e9521ed259c-kube-api-access-dpxxg\") pod \"network-check-target-h6crz\" (UID: \"6e8b3941-ab89-4838-bf55-6e9521ed259c\") " pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:01:10.990614 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990560    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0181bad8-3459-4c64-8e31-12395fd4c6c7-dbus\") pod \"global-pull-secret-syncer-x8wrn\" (UID: \"0181bad8-3459-4c64-8e31-12395fd4c6c7\") " pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:01:10.990614 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990584    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-systemd-units\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:10.991059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990646    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/626900f3-1832-46a5-8c82-70d09284614c-env-overrides\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:10.991059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990675    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-hostroot\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:10.991059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990718    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-cni-binary-copy\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:10.991059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990728    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g525b\" (UniqueName: \"kubernetes.io/projected/153d1ce3-22c2-444c-9bd2-ee91ff044d29-kube-api-access-g525b\") pod \"node-resolver-g2rp5\" (UID: \"153d1ce3-22c2-444c-9bd2-ee91ff044d29\") " pod="openshift-dns/node-resolver-g2rp5"
Nov 03 20:01:10.991059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990785    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-systemd\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:10.991059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990828    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/626900f3-1832-46a5-8c82-70d09284614c-env-overrides\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:10.991059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990878    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/626900f3-1832-46a5-8c82-70d09284614c-ovn-node-metrics-cert\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:10.991059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990913    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-multus-socket-dir-parent\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:10.991059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990947    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-var-lib-cni-bin\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:10.991059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990957    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 20:01:10.991059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990980    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-sysctl-conf\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:10.991059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.990984    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 20:01:10.991465 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.991085    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/626900f3-1832-46a5-8c82-70d09284614c-ovn-node-metrics-cert\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:10.991465 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.991102    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-system-cni-dir\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:10.991465 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.991125    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-cni-bin\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:10.991465 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.991172    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 20:01:10.991465 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.991184    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 20:01:10.991465 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.991167    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-multus-conf-dir\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:10.991465 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.991197    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 20:01:10.991465 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.991213    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xg4bg\""
Nov 03 20:01:10.991465 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.991236    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-lib-modules\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:10.991465 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.991236    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 20:01:10.991465 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.991281    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 20:01:10.991465 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.991283    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78003043-36f1-452a-8ccf-37b574877715-host\") pod \"node-ca-tt4df\" (UID: \"78003043-36f1-452a-8ccf-37b574877715\") " pod="openshift-image-registry/node-ca-tt4df"
Nov 03 20:01:10.998726 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.998711    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv682\" (UniqueName: \"kubernetes.io/projected/78003043-36f1-452a-8ccf-37b574877715-kube-api-access-gv682\") pod \"node-ca-tt4df\" (UID: \"78003043-36f1-452a-8ccf-37b574877715\") " pod="openshift-image-registry/node-ca-tt4df"
Nov 03 20:01:10.999014 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.999000    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjlck\" (UniqueName: \"kubernetes.io/projected/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-kube-api-access-tjlck\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:10.999176 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.999159    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mlgw\" (UniqueName: \"kubernetes.io/projected/c5965c2f-be74-431c-a8ff-86598070072a-kube-api-access-6mlgw\") pod \"iptables-alerter-v9qll\" (UID: \"c5965c2f-be74-431c-a8ff-86598070072a\") " pod="openshift-network-operator/iptables-alerter-v9qll"
Nov 03 20:01:10.999495 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.999475    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkrkh\" (UniqueName: \"kubernetes.io/projected/626900f3-1832-46a5-8c82-70d09284614c-kube-api-access-pkrkh\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:10.999570 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.999553    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b8s7\" (UniqueName: \"kubernetes.io/projected/368fd902-a222-4a5b-9bbf-a8decb45ad31-kube-api-access-7b8s7\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:10.999986 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:10.999969    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k9l5\" (UniqueName: \"kubernetes.io/projected/3492be51-5e92-4369-8fc5-85d5ce0523fd-kube-api-access-5k9l5\") pod \"network-metrics-daemon-286gh\" (UID: \"3492be51-5e92-4369-8fc5-85d5ce0523fd\") " pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:01:11.000127 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.000110    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g525b\" (UniqueName: \"kubernetes.io/projected/153d1ce3-22c2-444c-9bd2-ee91ff044d29-kube-api-access-g525b\") pod \"node-resolver-g2rp5\" (UID: \"153d1ce3-22c2-444c-9bd2-ee91ff044d29\") " pod="openshift-dns/node-resolver-g2rp5"
Nov 03 20:01:11.000389 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.000370    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpxxg\" (UniqueName: \"kubernetes.io/projected/6e8b3941-ab89-4838-bf55-6e9521ed259c-kube-api-access-dpxxg\") pod \"network-check-target-h6crz\" (UID: \"6e8b3941-ab89-4838-bf55-6e9521ed259c\") " pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:01:11.046545 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.046528    4421 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:01:11.046718 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.046703    4421 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:01:11.057702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.057681    4421 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:01:11.057819 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.057716    4421 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 20:01:11.057819 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:01:11.057738    4421 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:01:11.057819 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:01:11.057763    4421 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal"
Nov 03 20:01:11.077606 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.077584    4421 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 20:01:11.091888 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.091865    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls\") pod \"dns-default-wb9sn\" (UID: \"eaaa35b2-aee2-40f8-956b-0b9f697dc994\") " pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:11.091988 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.091891    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d61c96a-2363-47b2-949a-300d9b084779-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:01:11.091988 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.091916    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9d61c96a-2363-47b2-949a-300d9b084779-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:01:11.091988 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.091932    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-node-log\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.092078 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.091996    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-run-ovn-kubernetes\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.092078 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092029    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-kubernetes\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:11.092078 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092032    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-run-ovn-kubernetes\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.092078 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092046    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/153d1ce3-22c2-444c-9bd2-ee91ff044d29-hosts-file\") pod \"node-resolver-g2rp5\" (UID: \"153d1ce3-22c2-444c-9bd2-ee91ff044d29\") " pod="openshift-dns/node-resolver-g2rp5"
Nov 03 20:01:11.092078 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092071    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-etc-openvswitch\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.092323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092005    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-node-log\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.092323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092093    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/153d1ce3-22c2-444c-9bd2-ee91ff044d29-hosts-file\") pod \"node-resolver-g2rp5\" (UID: \"153d1ce3-22c2-444c-9bd2-ee91ff044d29\") " pod="openshift-dns/node-resolver-g2rp5"
Nov 03 20:01:11.092323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092095    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-run-k8s-cni-cncf-io\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:11.092323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092124    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3492be51-5e92-4369-8fc5-85d5ce0523fd-metrics-certs\") pod \"network-metrics-daemon-286gh\" (UID: \"3492be51-5e92-4369-8fc5-85d5ce0523fd\") " pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:01:11.092323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092224    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-kubernetes\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:11.092323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092220    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-run-systemd\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.092323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092263    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-run-systemd\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.092323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092270    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-host\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:11.092323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092298    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0181bad8-3459-4c64-8e31-12395fd4c6c7-dbus\") pod \"global-pull-secret-syncer-x8wrn\" (UID: \"0181bad8-3459-4c64-8e31-12395fd4c6c7\") " pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:01:11.092323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092320    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-host\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:11.092323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092322    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-etc-openvswitch\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.092832 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092354    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9d61c96a-2363-47b2-949a-300d9b084779-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:01:11.092832 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092325    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrkkm\" (UniqueName: \"kubernetes.io/projected/71697994-0833-4570-83d3-afa5739725e9-kube-api-access-rrkkm\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:01:11.092832 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092416    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-run-k8s-cni-cncf-io\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:11.092832 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092414    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:11.092832 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092484    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-systemd-units\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.092832 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092533    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-hostroot\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:11.092832 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092553    4421 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 20:01:11.092832 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092560    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/07e7cf29-701a-4c9d-a3ad-12d4846157c2-agent-certs\") pod \"konnectivity-agent-k5b5s\" (UID: \"07e7cf29-701a-4c9d-a3ad-12d4846157c2\") " pod="kube-system/konnectivity-agent-k5b5s"
Nov 03 20:01:11.092832 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092644    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0181bad8-3459-4c64-8e31-12395fd4c6c7-dbus\") pod \"global-pull-secret-syncer-x8wrn\" (UID: \"0181bad8-3459-4c64-8e31-12395fd4c6c7\") " pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:01:11.092832 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092676    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d61c96a-2363-47b2-949a-300d9b084779-system-cni-dir\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:01:11.092832 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092704    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:11.092832 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092727    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-systemd\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:11.092832 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092753    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-multus-socket-dir-parent\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:11.092832 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092775    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-var-lib-cni-bin\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:11.092832 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092799    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-sysctl-conf\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:11.093512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092845    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-systemd-units\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.093512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092848    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-system-cni-dir\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:11.093512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092890    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-system-cni-dir\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:11.093512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092927    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzfmw\" (UniqueName: \"kubernetes.io/projected/eaaa35b2-aee2-40f8-956b-0b9f697dc994-kube-api-access-qzfmw\") pod \"dns-default-wb9sn\" (UID: \"eaaa35b2-aee2-40f8-956b-0b9f697dc994\") " pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:11.093512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092971    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/07e7cf29-701a-4c9d-a3ad-12d4846157c2-agent-certs\") pod \"konnectivity-agent-k5b5s\" (UID: \"07e7cf29-701a-4c9d-a3ad-12d4846157c2\") " pod="kube-system/konnectivity-agent-k5b5s"
Nov 03 20:01:11.093512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092982    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/78748ccd-0718-470e-ade1-74ff636db6b7-signing-key\") pod \"service-ca-5fffc97bdf-brm42\" (UID: \"78748ccd-0718-470e-ade1-74ff636db6b7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brm42"
Nov 03 20:01:11.093512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.092996    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-hostroot\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:11.093512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.093306    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:11.093512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.093310    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/78748ccd-0718-470e-ade1-74ff636db6b7-signing-cabundle\") pod \"service-ca-5fffc97bdf-brm42\" (UID: \"78748ccd-0718-470e-ade1-74ff636db6b7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brm42"
Nov 03 20:01:11.093512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.093049    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-multus-socket-dir-parent\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:11.093512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.093092    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-var-lib-cni-bin\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:11.093512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.093013    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/78748ccd-0718-470e-ade1-74ff636db6b7-signing-cabundle\") pod \"service-ca-5fffc97bdf-brm42\" (UID: \"78748ccd-0718-470e-ade1-74ff636db6b7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brm42"
Nov 03 20:01:11.093512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.093094    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-systemd\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:11.093512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.093360    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n79bw\" (UniqueName: \"kubernetes.io/projected/126ae513-97e3-4ef0-b415-5ec03ea09960-kube-api-access-n79bw\") pod \"ingress-canary-sx9rt\" (UID: \"126ae513-97e3-4ef0-b415-5ec03ea09960\") " pod="openshift-ingress-canary/ingress-canary-sx9rt"
Nov 03 20:01:11.093512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.093385    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:01:11.093512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.093418    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-cni-bin\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.093512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.093473    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-multus-conf-dir\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:11.093512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.093518    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-lib-modules\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.093572    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78003043-36f1-452a-8ccf-37b574877715-host\") pod \"node-ca-tt4df\" (UID: \"78003043-36f1-452a-8ccf-37b574877715\") " pod="openshift-image-registry/node-ca-tt4df"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.093610    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/126ae513-97e3-4ef0-b415-5ec03ea09960-cert\") pod \"ingress-canary-sx9rt\" (UID: \"126ae513-97e3-4ef0-b415-5ec03ea09960\") " pod="openshift-ingress-canary/ingress-canary-sx9rt"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.093627    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-cni-bin\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.093635    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.093673    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-var-lib-kubelet\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.093761    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-lib-modules\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.093786    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-multus-conf-dir\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.093804    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-var-lib-kubelet\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.093822    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78003043-36f1-452a-8ccf-37b574877715-host\") pod \"node-ca-tt4df\" (UID: \"78003043-36f1-452a-8ccf-37b574877715\") " pod="openshift-image-registry/node-ca-tt4df"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.093859    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-sysctl-conf\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.093886    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-modprobe-d\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.093911    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/78748ccd-0718-470e-ade1-74ff636db6b7-signing-key\") pod \"service-ca-5fffc97bdf-brm42\" (UID: \"78748ccd-0718-470e-ade1-74ff636db6b7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brm42"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.093967    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/126ae513-97e3-4ef0-b415-5ec03ea09960-cert\") pod \"ingress-canary-sx9rt\" (UID: \"126ae513-97e3-4ef0-b415-5ec03ea09960\") " pod="openshift-ingress-canary/ingress-canary-sx9rt"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.093966    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5965c2f-be74-431c-a8ff-86598070072a-host-slash\") pod \"iptables-alerter-v9qll\" (UID: \"c5965c2f-be74-431c-a8ff-86598070072a\") " pod="openshift-network-operator/iptables-alerter-v9qll"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094004    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-modprobe-d\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094008    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/07e7cf29-701a-4c9d-a3ad-12d4846157c2-konnectivity-ca\") pod \"konnectivity-agent-k5b5s\" (UID: \"07e7cf29-701a-4c9d-a3ad-12d4846157c2\") " pod="kube-system/konnectivity-agent-k5b5s"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094036    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-socket-dir\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094070    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-sys-fs\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094083    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5965c2f-be74-431c-a8ff-86598070072a-host-slash\") pod \"iptables-alerter-v9qll\" (UID: \"c5965c2f-be74-431c-a8ff-86598070072a\") " pod="openshift-network-operator/iptables-alerter-v9qll"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094129    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-slash\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094146    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-cni-netd\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094183    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-slash\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094191    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/07e7cf29-701a-4c9d-a3ad-12d4846157c2-konnectivity-ca\") pod \"konnectivity-agent-k5b5s\" (UID: \"07e7cf29-701a-4c9d-a3ad-12d4846157c2\") " pod="kube-system/konnectivity-agent-k5b5s"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094195    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-cni-netd\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094204    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaaa35b2-aee2-40f8-956b-0b9f697dc994-tmp-dir\") pod \"dns-default-wb9sn\" (UID: \"eaaa35b2-aee2-40f8-956b-0b9f697dc994\") " pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094235    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-crio-socket\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094262    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eaaa35b2-aee2-40f8-956b-0b9f697dc994-tmp-dir\") pod \"dns-default-wb9sn\" (UID: \"eaaa35b2-aee2-40f8-956b-0b9f697dc994\") " pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:11.094267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094261    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-552wl\" (UniqueName: \"kubernetes.io/projected/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-kube-api-access-552wl\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094331    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-var-lib-openvswitch\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094370    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-var-lib-openvswitch\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094400    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-sysctl-d\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094420    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-var-lib-kubelet\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094437    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9d61c96a-2363-47b2-949a-300d9b084779-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094470    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-data-volume\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094492    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-var-lib-kubelet\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094525    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-run-openvswitch\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094549    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-data-volume\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094557    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-run-openvswitch\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094527    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-sysctl-d\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094557    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-sys\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094630    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtcnf\" (UniqueName: \"kubernetes.io/projected/78748ccd-0718-470e-ade1-74ff636db6b7-kube-api-access-gtcnf\") pod \"service-ca-5fffc97bdf-brm42\" (UID: \"78748ccd-0718-470e-ade1-74ff636db6b7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brm42"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094641    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-sys\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094655    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-kubelet\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094675    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9d61c96a-2363-47b2-949a-300d9b084779-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094682    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094724    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-run-netns\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094726    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094739    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-kubelet\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094749    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-cnibin\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094768    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-host-run-netns\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094771    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-var-lib-cni-multus\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094805    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-var-lib-cni-multus\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094805    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-etc-kubernetes\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094815    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-cnibin\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094836    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-sysconfig\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094837    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-etc-kubernetes\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094879    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-etc-sysconfig\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094901    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-run-ovn\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094924    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-multus-cni-dir\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.094950    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-run-netns\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.095026    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-run-netns\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.095052    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-multus-cni-dir\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.095058    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0181bad8-3459-4c64-8e31-12395fd4c6c7-kubelet-config\") pod \"global-pull-secret-syncer-x8wrn\" (UID: \"0181bad8-3459-4c64-8e31-12395fd4c6c7\") " pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.095085    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eaaa35b2-aee2-40f8-956b-0b9f697dc994-config-volume\") pod \"dns-default-wb9sn\" (UID: \"eaaa35b2-aee2-40f8-956b-0b9f697dc994\") " pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.095054    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-run-ovn\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.095109    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d61c96a-2363-47b2-949a-300d9b084779-cnibin\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.095160    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0181bad8-3459-4c64-8e31-12395fd4c6c7-kubelet-config\") pod \"global-pull-secret-syncer-x8wrn\" (UID: \"0181bad8-3459-4c64-8e31-12395fd4c6c7\") " pod="kube-system/global-pull-secret-syncer-x8wrn"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.095233    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-device-dir\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.095265    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-log-socket\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.095288    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-run\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.095293    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3492be51-5e92-4369-8fc5-85d5ce0523fd-metrics-certs\") pod \"network-metrics-daemon-286gh\" (UID: \"3492be51-5e92-4369-8fc5-85d5ce0523fd\") " pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.095311    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d61c96a-2363-47b2-949a-300d9b084779-cni-binary-copy\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.095327    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/eaaa35b2-aee2-40f8-956b-0b9f697dc994-config-volume\") pod \"dns-default-wb9sn\" (UID: \"eaaa35b2-aee2-40f8-956b-0b9f697dc994\") " pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.095336    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/626900f3-1832-46a5-8c82-70d09284614c-log-socket\") pod \"ovnkube-node-lcp7r\" (UID: \"626900f3-1832-46a5-8c82-70d09284614c\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.095360    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/368fd902-a222-4a5b-9bbf-a8decb45ad31-run\") pod \"tuned-w72hd\" (UID: \"368fd902-a222-4a5b-9bbf-a8decb45ad31\") " pod="openshift-cluster-node-tuning-operator/tuned-w72hd"
Nov 03 20:01:11.095343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.095337    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5q9m\" (UniqueName: \"kubernetes.io/projected/9d61c96a-2363-47b2-949a-300d9b084779-kube-api-access-j5q9m\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:01:11.097552 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.095401    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-registration-dir\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:01:11.097552 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.095420    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-run-multus-certs\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:11.097552 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.095500    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d61c96a-2363-47b2-949a-300d9b084779-cni-binary-copy\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:01:11.097552 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.095533    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac4c4fc1-a4c5-4f00-97bc-b201eede1226-host-run-multus-certs\") pod \"multus-mq5xz\" (UID: \"ac4c4fc1-a4c5-4f00-97bc-b201eede1226\") " pod="openshift-multus/multus-mq5xz"
Nov 03 20:01:11.102872 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.102850    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrkkm\" (UniqueName: \"kubernetes.io/projected/71697994-0833-4570-83d3-afa5739725e9-kube-api-access-rrkkm\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:01:11.103571 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.103553    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzfmw\" (UniqueName: \"kubernetes.io/projected/eaaa35b2-aee2-40f8-956b-0b9f697dc994-kube-api-access-qzfmw\") pod \"dns-default-wb9sn\" (UID: \"eaaa35b2-aee2-40f8-956b-0b9f697dc994\") " pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:11.103819 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.103804    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n79bw\" (UniqueName: \"kubernetes.io/projected/126ae513-97e3-4ef0-b415-5ec03ea09960-kube-api-access-n79bw\") pod \"ingress-canary-sx9rt\" (UID: \"126ae513-97e3-4ef0-b415-5ec03ea09960\") " pod="openshift-ingress-canary/ingress-canary-sx9rt"
Nov 03 20:01:11.104169 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.104150    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-552wl\" (UniqueName: \"kubernetes.io/projected/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-kube-api-access-552wl\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:11.104491 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.104475    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtcnf\" (UniqueName: \"kubernetes.io/projected/78748ccd-0718-470e-ade1-74ff636db6b7-kube-api-access-gtcnf\") pod \"service-ca-5fffc97bdf-brm42\" (UID: \"78748ccd-0718-470e-ade1-74ff636db6b7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-brm42"
Nov 03 20:01:11.104996 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.104976    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5q9m\" (UniqueName: \"kubernetes.io/projected/9d61c96a-2363-47b2-949a-300d9b084779-kube-api-access-j5q9m\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:01:11.196324 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.196296    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d61c96a-2363-47b2-949a-300d9b084779-cnibin\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:01:11.196324 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.196324    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-device-dir\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:01:11.196471 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.196339    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-registration-dir\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:01:11.196471 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.196356    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls\") pod \"dns-default-wb9sn\" (UID: \"eaaa35b2-aee2-40f8-956b-0b9f697dc994\") " pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:11.196471 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.196414    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d61c96a-2363-47b2-949a-300d9b084779-cnibin\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:01:11.196471 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.196425    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-registration-dir\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:01:11.196657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.196471    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d61c96a-2363-47b2-949a-300d9b084779-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:01:11.196657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.196509    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:11.196657 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:01:11.196502    4421 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 20:01:11.196657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.196530    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d61c96a-2363-47b2-949a-300d9b084779-system-cni-dir\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:01:11.196657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.196551    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:01:11.196657 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:01:11.196586    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls podName:eaaa35b2-aee2-40f8-956b-0b9f697dc994 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:11.696567231 +0000 UTC m=+1.829695694 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls") pod "dns-default-wb9sn" (UID: "eaaa35b2-aee2-40f8-956b-0b9f697dc994") : secret "dns-default-metrics-tls" not found
Nov 03 20:01:11.196657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.196591    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d61c96a-2363-47b2-949a-300d9b084779-tuning-conf-dir\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:01:11.196657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.196622    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d61c96a-2363-47b2-949a-300d9b084779-system-cni-dir\") pod \"multus-additional-cni-plugins-86vz2\" (UID: \"9d61c96a-2363-47b2-949a-300d9b084779\") " pod="openshift-multus/multus-additional-cni-plugins-86vz2"
Nov 03 20:01:11.196657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.196630    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:01:11.196657 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:01:11.196627    4421 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 20:01:11.196657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.196661    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:01:11.197008 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.196688    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-socket-dir\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:01:11.197008 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:01:11.196705    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-insights-runtime-extractor-tls podName:20b2d254-c5b2-4a59-8ce9-49a9f2c0f435 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:11.69668641 +0000 UTC m=+1.829814854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-insights-runtime-extractor-tls") pod "insights-runtime-extractor-74j6n" (UID: "20b2d254-c5b2-4a59-8ce9-49a9f2c0f435") : secret "insights-runtime-extractor-tls" not found
Nov 03 20:01:11.197008 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.196731    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-sys-fs\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:01:11.197008 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.196735    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-socket-dir\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:01:11.197008 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.196760    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-crio-socket\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:11.197008 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.196802    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-sys-fs\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:01:11.197008 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.196816    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:01:11.197008 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.196832    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-crio-socket\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:11.197008 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.196886    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/71697994-0833-4570-83d3-afa5739725e9-device-dir\") pod \"aws-ebs-csi-driver-node-7g7sb\" (UID: \"71697994-0833-4570-83d3-afa5739725e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7g7sb"
Nov 03 20:01:11.277796 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.276424    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-286gh"
Nov 03 20:01:11.592786 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.592722    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-286gh"]
Nov 03 20:01:11.595821 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:11.595792    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3492be51_5e92_4369_8fc5_85d5ce0523fd.slice/crio-a0eaf4bcc4f05bd90273fb3927ac68f9bbb658e6c11cda9881956c7ee475f7de WatchSource:0}: Error finding container a0eaf4bcc4f05bd90273fb3927ac68f9bbb658e6c11cda9881956c7ee475f7de: Status 404 returned error can't find the container with id a0eaf4bcc4f05bd90273fb3927ac68f9bbb658e6c11cda9881956c7ee475f7de
Nov 03 20:01:11.597947 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.597923    4421 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:01:11.699173 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.699144    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls\") pod \"dns-default-wb9sn\" (UID: \"eaaa35b2-aee2-40f8-956b-0b9f697dc994\") " pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:11.699275 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.699186    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:11.699310 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:01:11.699286    4421 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 20:01:11.699310 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:01:11.699285    4421 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 20:01:11.699378 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:01:11.699345    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls podName:eaaa35b2-aee2-40f8-956b-0b9f697dc994 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:12.699331427 +0000 UTC m=+2.832459880 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls") pod "dns-default-wb9sn" (UID: "eaaa35b2-aee2-40f8-956b-0b9f697dc994") : secret "dns-default-metrics-tls" not found
Nov 03 20:01:11.699378 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:01:11.699359    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-insights-runtime-extractor-tls podName:20b2d254-c5b2-4a59-8ce9-49a9f2c0f435 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:12.69935258 +0000 UTC m=+2.832481020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-insights-runtime-extractor-tls") pod "insights-runtime-extractor-74j6n" (UID: "20b2d254-c5b2-4a59-8ce9-49a9f2c0f435") : secret "insights-runtime-extractor-tls" not found
Nov 03 20:01:11.763765 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:11.763736    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:01:12.050169 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:12.050138    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-286gh" event={"ID":"3492be51-5e92-4369-8fc5-85d5ce0523fd","Type":"ContainerStarted","Data":"a0eaf4bcc4f05bd90273fb3927ac68f9bbb658e6c11cda9881956c7ee475f7de"}
Nov 03 20:01:12.707198 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:12.707166    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:12.707356 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:12.707225    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls\") pod \"dns-default-wb9sn\" (UID: \"eaaa35b2-aee2-40f8-956b-0b9f697dc994\") " pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:12.707356 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:01:12.707337    4421 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 20:01:12.707476 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:01:12.707386    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls podName:eaaa35b2-aee2-40f8-956b-0b9f697dc994 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:14.707374545 +0000 UTC m=+4.840502998 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls") pod "dns-default-wb9sn" (UID: "eaaa35b2-aee2-40f8-956b-0b9f697dc994") : secret "dns-default-metrics-tls" not found
Nov 03 20:01:12.709649 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:12.709627    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/20b2d254-c5b2-4a59-8ce9-49a9f2c0f435-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-74j6n\" (UID: \"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435\") " pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:12.781977 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:12.781941    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-74j6n"
Nov 03 20:01:13.055809 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:13.055784    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-286gh" event={"ID":"3492be51-5e92-4369-8fc5-85d5ce0523fd","Type":"ContainerStarted","Data":"fcfdbf2dced0806b6b69df465edded919c2db10a9c8f98c5252177bd02c00ece"}
Nov 03 20:01:13.198757 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:13.198730    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-74j6n"]
Nov 03 20:01:13.201614 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:13.201576    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b2d254_c5b2_4a59_8ce9_49a9f2c0f435.slice/crio-e0ff34393e3bc22004bfc31d1d4608ec8af7d514c34e75d773f998538f275ff9 WatchSource:0}: Error finding container e0ff34393e3bc22004bfc31d1d4608ec8af7d514c34e75d773f998538f275ff9: Status 404 returned error can't find the container with id e0ff34393e3bc22004bfc31d1d4608ec8af7d514c34e75d773f998538f275ff9
Nov 03 20:01:14.064438 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:14.064400    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-286gh" event={"ID":"3492be51-5e92-4369-8fc5-85d5ce0523fd","Type":"ContainerStarted","Data":"32c223f3127d812a33f1cf68cbb8aca354cd1597e6aa5638f1cdf3a8c591def7"}
Nov 03 20:01:14.066179 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:14.066153    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-74j6n" event={"ID":"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435","Type":"ContainerStarted","Data":"f34237106de353c51bc594a151fa8ef4261cad29c102caab4a90bd92f9f190c7"}
Nov 03 20:01:14.066309 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:14.066193    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-74j6n" event={"ID":"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435","Type":"ContainerStarted","Data":"e0ff34393e3bc22004bfc31d1d4608ec8af7d514c34e75d773f998538f275ff9"}
Nov 03 20:01:14.720757 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:14.720721    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls\") pod \"dns-default-wb9sn\" (UID: \"eaaa35b2-aee2-40f8-956b-0b9f697dc994\") " pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:14.723169 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:14.723146    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eaaa35b2-aee2-40f8-956b-0b9f697dc994-metrics-tls\") pod \"dns-default-wb9sn\" (UID: \"eaaa35b2-aee2-40f8-956b-0b9f697dc994\") " pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:14.880494 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:14.880459    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:15.070243 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:15.070159    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-74j6n" event={"ID":"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435","Type":"ContainerStarted","Data":"8eb3c1f75d85d6444c764a249b72477f6bf3e2ac6942406b76584bfb6d5994db"}
Nov 03 20:01:15.137971 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:15.137948    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:15.151770 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:15.151744    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wb9sn"]
Nov 03 20:01:15.154774 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:15.154745    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaaa35b2_aee2_40f8_956b_0b9f697dc994.slice/crio-e3b8bebb5e65e29b38d308a85c9b6c4823e7f7c7d8349550f1dd2754c805f50c WatchSource:0}: Error finding container e3b8bebb5e65e29b38d308a85c9b6c4823e7f7c7d8349550f1dd2754c805f50c: Status 404 returned error can't find the container with id e3b8bebb5e65e29b38d308a85c9b6c4823e7f7c7d8349550f1dd2754c805f50c
Nov 03 20:01:16.077987 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:16.077953    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wb9sn" event={"ID":"eaaa35b2-aee2-40f8-956b-0b9f697dc994","Type":"ContainerStarted","Data":"e3b8bebb5e65e29b38d308a85c9b6c4823e7f7c7d8349550f1dd2754c805f50c"}
Nov 03 20:01:16.080288 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:16.080245    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-74j6n" event={"ID":"20b2d254-c5b2-4a59-8ce9-49a9f2c0f435","Type":"ContainerStarted","Data":"da9f99396d2fce5c10e959ef2ad5e3a1fc6c5f09cdee2b9314f5c18110387f30"}
Nov 03 20:01:18.970283 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:18.970249    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k5b5s"
Nov 03 20:01:19.080870 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:19.080842    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:19.081052 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:19.081040    4421 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 20:01:19.636751 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:01:19.636717    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:01:20.095131 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:20.095090    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wb9sn" event={"ID":"eaaa35b2-aee2-40f8-956b-0b9f697dc994","Type":"ContainerStarted","Data":"c65135cc9ea2fa541d230c1e26a6d463dd44e264f9adae216a53cc9180eb6da5"}
Nov 03 20:01:20.095541 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:20.095162    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wb9sn" event={"ID":"eaaa35b2-aee2-40f8-956b-0b9f697dc994","Type":"ContainerStarted","Data":"9791c1ea670595a43da2515f147ee781bd850388e409050e0d0296a3c1c2251d"}
Nov 03 20:01:20.095541 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:20.095256    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:21.104418 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.104388    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7679cb5584-qgtd8"]
Nov 03 20:01:21.106399 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.106383    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:21.122413 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.122390    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7679cb5584-qgtd8"]
Nov 03 20:01:21.123020 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.122991    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 20:01:21.123128 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.123055    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 20:01:21.123128 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.123069    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 20:01:21.123270 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.123254    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pw8jf\""
Nov 03 20:01:21.131763 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.129894    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 20:01:21.160912 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.160887    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8953fd4e-dc26-4422-82bb-1c2e6fe569b5-image-registry-private-configuration\") pod \"image-registry-7679cb5584-qgtd8\" (UID: \"8953fd4e-dc26-4422-82bb-1c2e6fe569b5\") " pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:21.161016 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.160921    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8953fd4e-dc26-4422-82bb-1c2e6fe569b5-ca-trust-extracted\") pod \"image-registry-7679cb5584-qgtd8\" (UID: \"8953fd4e-dc26-4422-82bb-1c2e6fe569b5\") " pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:21.161016 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.160946    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8953fd4e-dc26-4422-82bb-1c2e6fe569b5-registry-certificates\") pod \"image-registry-7679cb5584-qgtd8\" (UID: \"8953fd4e-dc26-4422-82bb-1c2e6fe569b5\") " pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:21.161016 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.160970    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8953fd4e-dc26-4422-82bb-1c2e6fe569b5-bound-sa-token\") pod \"image-registry-7679cb5584-qgtd8\" (UID: \"8953fd4e-dc26-4422-82bb-1c2e6fe569b5\") " pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:21.161191 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.161171    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q9wg\" (UniqueName: \"kubernetes.io/projected/8953fd4e-dc26-4422-82bb-1c2e6fe569b5-kube-api-access-5q9wg\") pod \"image-registry-7679cb5584-qgtd8\" (UID: \"8953fd4e-dc26-4422-82bb-1c2e6fe569b5\") " pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:21.161256 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.161236    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8953fd4e-dc26-4422-82bb-1c2e6fe569b5-registry-tls\") pod \"image-registry-7679cb5584-qgtd8\" (UID: \"8953fd4e-dc26-4422-82bb-1c2e6fe569b5\") " pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:21.161295 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.161266    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8953fd4e-dc26-4422-82bb-1c2e6fe569b5-trusted-ca\") pod \"image-registry-7679cb5584-qgtd8\" (UID: \"8953fd4e-dc26-4422-82bb-1c2e6fe569b5\") " pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:21.161295 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.161286    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8953fd4e-dc26-4422-82bb-1c2e6fe569b5-installation-pull-secrets\") pod \"image-registry-7679cb5584-qgtd8\" (UID: \"8953fd4e-dc26-4422-82bb-1c2e6fe569b5\") " pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:21.261637 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.261609    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8953fd4e-dc26-4422-82bb-1c2e6fe569b5-installation-pull-secrets\") pod \"image-registry-7679cb5584-qgtd8\" (UID: \"8953fd4e-dc26-4422-82bb-1c2e6fe569b5\") " pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:21.261801 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.261653    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8953fd4e-dc26-4422-82bb-1c2e6fe569b5-image-registry-private-configuration\") pod \"image-registry-7679cb5584-qgtd8\" (UID: \"8953fd4e-dc26-4422-82bb-1c2e6fe569b5\") " pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:21.261843 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.261824    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8953fd4e-dc26-4422-82bb-1c2e6fe569b5-ca-trust-extracted\") pod \"image-registry-7679cb5584-qgtd8\" (UID: \"8953fd4e-dc26-4422-82bb-1c2e6fe569b5\") " pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:21.261882 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.261859    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8953fd4e-dc26-4422-82bb-1c2e6fe569b5-registry-certificates\") pod \"image-registry-7679cb5584-qgtd8\" (UID: \"8953fd4e-dc26-4422-82bb-1c2e6fe569b5\") " pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:21.261921 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.261884    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8953fd4e-dc26-4422-82bb-1c2e6fe569b5-bound-sa-token\") pod \"image-registry-7679cb5584-qgtd8\" (UID: \"8953fd4e-dc26-4422-82bb-1c2e6fe569b5\") " pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:21.261971 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.261916    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q9wg\" (UniqueName: \"kubernetes.io/projected/8953fd4e-dc26-4422-82bb-1c2e6fe569b5-kube-api-access-5q9wg\") pod \"image-registry-7679cb5584-qgtd8\" (UID: \"8953fd4e-dc26-4422-82bb-1c2e6fe569b5\") " pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:21.261971 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.261951    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8953fd4e-dc26-4422-82bb-1c2e6fe569b5-registry-tls\") pod \"image-registry-7679cb5584-qgtd8\" (UID: \"8953fd4e-dc26-4422-82bb-1c2e6fe569b5\") " pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:21.262064 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.261973    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8953fd4e-dc26-4422-82bb-1c2e6fe569b5-trusted-ca\") pod \"image-registry-7679cb5584-qgtd8\" (UID: \"8953fd4e-dc26-4422-82bb-1c2e6fe569b5\") " pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:21.262267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.262242    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8953fd4e-dc26-4422-82bb-1c2e6fe569b5-ca-trust-extracted\") pod \"image-registry-7679cb5584-qgtd8\" (UID: \"8953fd4e-dc26-4422-82bb-1c2e6fe569b5\") " pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:21.262836 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.262814    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8953fd4e-dc26-4422-82bb-1c2e6fe569b5-registry-certificates\") pod \"image-registry-7679cb5584-qgtd8\" (UID: \"8953fd4e-dc26-4422-82bb-1c2e6fe569b5\") " pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:21.263316 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.263292    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8953fd4e-dc26-4422-82bb-1c2e6fe569b5-trusted-ca\") pod \"image-registry-7679cb5584-qgtd8\" (UID: \"8953fd4e-dc26-4422-82bb-1c2e6fe569b5\") " pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:21.264372 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.264352    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8953fd4e-dc26-4422-82bb-1c2e6fe569b5-installation-pull-secrets\") pod \"image-registry-7679cb5584-qgtd8\" (UID: \"8953fd4e-dc26-4422-82bb-1c2e6fe569b5\") " pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:21.264502 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.264484    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8953fd4e-dc26-4422-82bb-1c2e6fe569b5-image-registry-private-configuration\") pod \"image-registry-7679cb5584-qgtd8\" (UID: \"8953fd4e-dc26-4422-82bb-1c2e6fe569b5\") " pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:21.264716 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.264699    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8953fd4e-dc26-4422-82bb-1c2e6fe569b5-registry-tls\") pod \"image-registry-7679cb5584-qgtd8\" (UID: \"8953fd4e-dc26-4422-82bb-1c2e6fe569b5\") " pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:21.276846 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.276817    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8953fd4e-dc26-4422-82bb-1c2e6fe569b5-bound-sa-token\") pod \"image-registry-7679cb5584-qgtd8\" (UID: \"8953fd4e-dc26-4422-82bb-1c2e6fe569b5\") " pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:21.278449 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.278430    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q9wg\" (UniqueName: \"kubernetes.io/projected/8953fd4e-dc26-4422-82bb-1c2e6fe569b5-kube-api-access-5q9wg\") pod \"image-registry-7679cb5584-qgtd8\" (UID: \"8953fd4e-dc26-4422-82bb-1c2e6fe569b5\") " pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:21.415775 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.415700    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:21.690396 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:21.690368    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7679cb5584-qgtd8"]
Nov 03 20:01:21.693453 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:21.693428    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8953fd4e_dc26_4422_82bb_1c2e6fe569b5.slice/crio-b2bd8727bb223b0050a3aaa994da16f1be45aed382a9de08a05619a376ba90be WatchSource:0}: Error finding container b2bd8727bb223b0050a3aaa994da16f1be45aed382a9de08a05619a376ba90be: Status 404 returned error can't find the container with id b2bd8727bb223b0050a3aaa994da16f1be45aed382a9de08a05619a376ba90be
Nov 03 20:01:22.102737 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:22.102641    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7679cb5584-qgtd8" event={"ID":"8953fd4e-dc26-4422-82bb-1c2e6fe569b5","Type":"ContainerStarted","Data":"6602a91597fd57b833133b8d9df726f4f9280499296df80d6885bac157e3db7c"}
Nov 03 20:01:22.102737 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:22.102705    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7679cb5584-qgtd8" event={"ID":"8953fd4e-dc26-4422-82bb-1c2e6fe569b5","Type":"ContainerStarted","Data":"b2bd8727bb223b0050a3aaa994da16f1be45aed382a9de08a05619a376ba90be"}
Nov 03 20:01:22.102956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:22.102751    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:25.502385 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:25.502359    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcp7r"
Nov 03 20:01:25.537120 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:25.537066    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7679cb5584-qgtd8" podStartSLOduration=4.537047223 podStartE2EDuration="4.537047223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:22.128892567 +0000 UTC m=+12.262021030" watchObservedRunningTime="2026-11-03 20:01:25.537047223 +0000 UTC m=+15.670175685"
Nov 03 20:01:30.100472 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:30.100441    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wb9sn"
Nov 03 20:01:36.553316 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:36.553286    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wb9sn_eaaa35b2-aee2-40f8-956b-0b9f697dc994/dns/0.log"
Nov 03 20:01:36.723805 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:36.723779    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wb9sn_eaaa35b2-aee2-40f8-956b-0b9f697dc994/kube-rbac-proxy/0.log"
Nov 03 20:01:36.929740 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:36.929661    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g2rp5_153d1ce3-22c2-444c-9bd2-ee91ff044d29/dns-node-resolver/0.log"
Nov 03 20:01:36.997733 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:36.997698    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-qsd4s"]
Nov 03 20:01:37.001397 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.001379    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qsd4s"
Nov 03 20:01:37.013379 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.013354    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-qsd4s"]
Nov 03 20:01:37.017895 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.017631    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 20:01:37.017895 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.017899    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 20:01:37.018095 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.017992    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 20:01:37.020656 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.020119    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 20:01:37.020656 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.020185    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-t7dgq\""
Nov 03 20:01:37.020656 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.020191    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 20:01:37.042619 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.042577    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw"]
Nov 03 20:01:37.045182 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.045165    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw"
Nov 03 20:01:37.065905 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.065881    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3971ef1f-1497-4eb4-b3a0-da2c5a75b619-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qsd4s\" (UID: \"3971ef1f-1497-4eb4-b3a0-da2c5a75b619\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qsd4s"
Nov 03 20:01:37.066016 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.065942    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3971ef1f-1497-4eb4-b3a0-da2c5a75b619-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qsd4s\" (UID: \"3971ef1f-1497-4eb4-b3a0-da2c5a75b619\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qsd4s"
Nov 03 20:01:37.066016 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.066008    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3971ef1f-1497-4eb4-b3a0-da2c5a75b619-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qsd4s\" (UID: \"3971ef1f-1497-4eb4-b3a0-da2c5a75b619\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qsd4s"
Nov 03 20:01:37.066116 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.066061    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9tqn\" (UniqueName: \"kubernetes.io/projected/3971ef1f-1497-4eb4-b3a0-da2c5a75b619-kube-api-access-z9tqn\") pod \"openshift-state-metrics-7bd7489957-qsd4s\" (UID: \"3971ef1f-1497-4eb4-b3a0-da2c5a75b619\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qsd4s"
Nov 03 20:01:37.072494 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.072467    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 03 20:01:37.072616 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.072522    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 03 20:01:37.072726 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.072703    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-f6sp2\""
Nov 03 20:01:37.073202 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.073186    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8b6ng"]
Nov 03 20:01:37.075651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.075633    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw"]
Nov 03 20:01:37.075750 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.075735    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.095291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.095266    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 03 20:01:37.095291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.095284    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 20:01:37.120163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.120137    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 20:01:37.120442 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.120428    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 20:01:37.120482 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.120450    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8ls9s\""
Nov 03 20:01:37.166997 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.166970    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/091b9d41-6f3f-4b38-b708-2ad312fc3af5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-k46fw\" (UID: \"091b9d41-6f3f-4b38-b708-2ad312fc3af5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw"
Nov 03 20:01:37.167116 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.167012    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9tqn\" (UniqueName: \"kubernetes.io/projected/3971ef1f-1497-4eb4-b3a0-da2c5a75b619-kube-api-access-z9tqn\") pod \"openshift-state-metrics-7bd7489957-qsd4s\" (UID: \"3971ef1f-1497-4eb4-b3a0-da2c5a75b619\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qsd4s"
Nov 03 20:01:37.167116 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.167041    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzd6q\" (UniqueName: \"kubernetes.io/projected/2b03c6cd-cfd6-4234-820d-d42854548f2d-kube-api-access-nzd6q\") pod \"node-exporter-8b6ng\" (UID: \"2b03c6cd-cfd6-4234-820d-d42854548f2d\") " pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.167116 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.167102    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/091b9d41-6f3f-4b38-b708-2ad312fc3af5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-k46fw\" (UID: \"091b9d41-6f3f-4b38-b708-2ad312fc3af5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw"
Nov 03 20:01:37.167263 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.167129    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/091b9d41-6f3f-4b38-b708-2ad312fc3af5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-k46fw\" (UID: \"091b9d41-6f3f-4b38-b708-2ad312fc3af5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw"
Nov 03 20:01:37.167263 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.167158    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2b03c6cd-cfd6-4234-820d-d42854548f2d-node-exporter-accelerators-collector-config\") pod \"node-exporter-8b6ng\" (UID: \"2b03c6cd-cfd6-4234-820d-d42854548f2d\") " pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.167263 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.167209    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/091b9d41-6f3f-4b38-b708-2ad312fc3af5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-k46fw\" (UID: \"091b9d41-6f3f-4b38-b708-2ad312fc3af5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw"
Nov 03 20:01:37.167263 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.167261    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b03c6cd-cfd6-4234-820d-d42854548f2d-metrics-client-ca\") pod \"node-exporter-8b6ng\" (UID: \"2b03c6cd-cfd6-4234-820d-d42854548f2d\") " pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.167380 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.167301    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2b03c6cd-cfd6-4234-820d-d42854548f2d-node-exporter-wtmp\") pod \"node-exporter-8b6ng\" (UID: \"2b03c6cd-cfd6-4234-820d-d42854548f2d\") " pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.167380 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.167328    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b03c6cd-cfd6-4234-820d-d42854548f2d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8b6ng\" (UID: \"2b03c6cd-cfd6-4234-820d-d42854548f2d\") " pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.167380 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.167351    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3971ef1f-1497-4eb4-b3a0-da2c5a75b619-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qsd4s\" (UID: \"3971ef1f-1497-4eb4-b3a0-da2c5a75b619\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qsd4s"
Nov 03 20:01:37.167380 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.167367    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3971ef1f-1497-4eb4-b3a0-da2c5a75b619-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qsd4s\" (UID: \"3971ef1f-1497-4eb4-b3a0-da2c5a75b619\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qsd4s"
Nov 03 20:01:37.167543 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.167480    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3971ef1f-1497-4eb4-b3a0-da2c5a75b619-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qsd4s\" (UID: \"3971ef1f-1497-4eb4-b3a0-da2c5a75b619\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qsd4s"
Nov 03 20:01:37.167543 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.167511    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2b03c6cd-cfd6-4234-820d-d42854548f2d-root\") pod \"node-exporter-8b6ng\" (UID: \"2b03c6cd-cfd6-4234-820d-d42854548f2d\") " pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.167663 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.167540    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b03c6cd-cfd6-4234-820d-d42854548f2d-sys\") pod \"node-exporter-8b6ng\" (UID: \"2b03c6cd-cfd6-4234-820d-d42854548f2d\") " pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.167698 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.167659    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2b03c6cd-cfd6-4234-820d-d42854548f2d-node-exporter-textfile\") pod \"node-exporter-8b6ng\" (UID: \"2b03c6cd-cfd6-4234-820d-d42854548f2d\") " pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.167741 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.167701    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2b03c6cd-cfd6-4234-820d-d42854548f2d-node-exporter-tls\") pod \"node-exporter-8b6ng\" (UID: \"2b03c6cd-cfd6-4234-820d-d42854548f2d\") " pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.167793 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.167733    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6ckv\" (UniqueName: \"kubernetes.io/projected/091b9d41-6f3f-4b38-b708-2ad312fc3af5-kube-api-access-w6ckv\") pod \"kube-state-metrics-5dc8d7d89b-k46fw\" (UID: \"091b9d41-6f3f-4b38-b708-2ad312fc3af5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw"
Nov 03 20:01:37.167867 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.167837    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/091b9d41-6f3f-4b38-b708-2ad312fc3af5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-k46fw\" (UID: \"091b9d41-6f3f-4b38-b708-2ad312fc3af5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw"
Nov 03 20:01:37.168317 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.168297    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3971ef1f-1497-4eb4-b3a0-da2c5a75b619-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qsd4s\" (UID: \"3971ef1f-1497-4eb4-b3a0-da2c5a75b619\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qsd4s"
Nov 03 20:01:37.169992 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.169970    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3971ef1f-1497-4eb4-b3a0-da2c5a75b619-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qsd4s\" (UID: \"3971ef1f-1497-4eb4-b3a0-da2c5a75b619\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qsd4s"
Nov 03 20:01:37.170077 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.170014    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3971ef1f-1497-4eb4-b3a0-da2c5a75b619-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qsd4s\" (UID: \"3971ef1f-1497-4eb4-b3a0-da2c5a75b619\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qsd4s"
Nov 03 20:01:37.179341 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.179319    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9tqn\" (UniqueName: \"kubernetes.io/projected/3971ef1f-1497-4eb4-b3a0-da2c5a75b619-kube-api-access-z9tqn\") pod \"openshift-state-metrics-7bd7489957-qsd4s\" (UID: \"3971ef1f-1497-4eb4-b3a0-da2c5a75b619\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qsd4s"
Nov 03 20:01:37.269120 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.269067    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/091b9d41-6f3f-4b38-b708-2ad312fc3af5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-k46fw\" (UID: \"091b9d41-6f3f-4b38-b708-2ad312fc3af5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw"
Nov 03 20:01:37.269120 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.269121    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzd6q\" (UniqueName: \"kubernetes.io/projected/2b03c6cd-cfd6-4234-820d-d42854548f2d-kube-api-access-nzd6q\") pod \"node-exporter-8b6ng\" (UID: \"2b03c6cd-cfd6-4234-820d-d42854548f2d\") " pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.269360 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.269141    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/091b9d41-6f3f-4b38-b708-2ad312fc3af5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-k46fw\" (UID: \"091b9d41-6f3f-4b38-b708-2ad312fc3af5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw"
Nov 03 20:01:37.269360 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.269159    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/091b9d41-6f3f-4b38-b708-2ad312fc3af5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-k46fw\" (UID: \"091b9d41-6f3f-4b38-b708-2ad312fc3af5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw"
Nov 03 20:01:37.269360 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.269214    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2b03c6cd-cfd6-4234-820d-d42854548f2d-node-exporter-accelerators-collector-config\") pod \"node-exporter-8b6ng\" (UID: \"2b03c6cd-cfd6-4234-820d-d42854548f2d\") " pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.269360 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.269270    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/091b9d41-6f3f-4b38-b708-2ad312fc3af5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-k46fw\" (UID: \"091b9d41-6f3f-4b38-b708-2ad312fc3af5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw"
Nov 03 20:01:37.269360 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.269307    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b03c6cd-cfd6-4234-820d-d42854548f2d-metrics-client-ca\") pod \"node-exporter-8b6ng\" (UID: \"2b03c6cd-cfd6-4234-820d-d42854548f2d\") " pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.269360 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.269338    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2b03c6cd-cfd6-4234-820d-d42854548f2d-node-exporter-wtmp\") pod \"node-exporter-8b6ng\" (UID: \"2b03c6cd-cfd6-4234-820d-d42854548f2d\") " pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.269702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.269362    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b03c6cd-cfd6-4234-820d-d42854548f2d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8b6ng\" (UID: \"2b03c6cd-cfd6-4234-820d-d42854548f2d\") " pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.269702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.269397    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2b03c6cd-cfd6-4234-820d-d42854548f2d-root\") pod \"node-exporter-8b6ng\" (UID: \"2b03c6cd-cfd6-4234-820d-d42854548f2d\") " pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.269702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.269421    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b03c6cd-cfd6-4234-820d-d42854548f2d-sys\") pod \"node-exporter-8b6ng\" (UID: \"2b03c6cd-cfd6-4234-820d-d42854548f2d\") " pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.269702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.269446    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2b03c6cd-cfd6-4234-820d-d42854548f2d-node-exporter-textfile\") pod \"node-exporter-8b6ng\" (UID: \"2b03c6cd-cfd6-4234-820d-d42854548f2d\") " pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.269702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.269475    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2b03c6cd-cfd6-4234-820d-d42854548f2d-node-exporter-tls\") pod \"node-exporter-8b6ng\" (UID: \"2b03c6cd-cfd6-4234-820d-d42854548f2d\") " pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.269702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.269504    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6ckv\" (UniqueName: \"kubernetes.io/projected/091b9d41-6f3f-4b38-b708-2ad312fc3af5-kube-api-access-w6ckv\") pod \"kube-state-metrics-5dc8d7d89b-k46fw\" (UID: \"091b9d41-6f3f-4b38-b708-2ad312fc3af5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw"
Nov 03 20:01:37.269702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.269521    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2b03c6cd-cfd6-4234-820d-d42854548f2d-root\") pod \"node-exporter-8b6ng\" (UID: \"2b03c6cd-cfd6-4234-820d-d42854548f2d\") " pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.269702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.269538    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/091b9d41-6f3f-4b38-b708-2ad312fc3af5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-k46fw\" (UID: \"091b9d41-6f3f-4b38-b708-2ad312fc3af5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw"
Nov 03 20:01:37.269702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.269682    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2b03c6cd-cfd6-4234-820d-d42854548f2d-node-exporter-wtmp\") pod \"node-exporter-8b6ng\" (UID: \"2b03c6cd-cfd6-4234-820d-d42854548f2d\") " pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.269702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.269697    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/091b9d41-6f3f-4b38-b708-2ad312fc3af5-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-k46fw\" (UID: \"091b9d41-6f3f-4b38-b708-2ad312fc3af5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw"
Nov 03 20:01:37.270197 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.269840    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2b03c6cd-cfd6-4234-820d-d42854548f2d-node-exporter-accelerators-collector-config\") pod \"node-exporter-8b6ng\" (UID: \"2b03c6cd-cfd6-4234-820d-d42854548f2d\") " pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.270197 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.269961    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/091b9d41-6f3f-4b38-b708-2ad312fc3af5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-k46fw\" (UID: \"091b9d41-6f3f-4b38-b708-2ad312fc3af5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw"
Nov 03 20:01:37.270197 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.269980    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2b03c6cd-cfd6-4234-820d-d42854548f2d-node-exporter-textfile\") pod \"node-exporter-8b6ng\" (UID: \"2b03c6cd-cfd6-4234-820d-d42854548f2d\") " pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.270197 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.269994    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/091b9d41-6f3f-4b38-b708-2ad312fc3af5-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-k46fw\" (UID: \"091b9d41-6f3f-4b38-b708-2ad312fc3af5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw"
Nov 03 20:01:37.270197 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.270037    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b03c6cd-cfd6-4234-820d-d42854548f2d-sys\") pod \"node-exporter-8b6ng\" (UID: \"2b03c6cd-cfd6-4234-820d-d42854548f2d\") " pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.270494 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.270474    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b03c6cd-cfd6-4234-820d-d42854548f2d-metrics-client-ca\") pod \"node-exporter-8b6ng\" (UID: \"2b03c6cd-cfd6-4234-820d-d42854548f2d\") " pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.272038 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.272014    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2b03c6cd-cfd6-4234-820d-d42854548f2d-node-exporter-tls\") pod \"node-exporter-8b6ng\" (UID: \"2b03c6cd-cfd6-4234-820d-d42854548f2d\") " pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.272147 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.272051    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/091b9d41-6f3f-4b38-b708-2ad312fc3af5-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-k46fw\" (UID: \"091b9d41-6f3f-4b38-b708-2ad312fc3af5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw"
Nov 03 20:01:37.272462 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.272439    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b03c6cd-cfd6-4234-820d-d42854548f2d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8b6ng\" (UID: \"2b03c6cd-cfd6-4234-820d-d42854548f2d\") " pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.272518 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.272500    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/091b9d41-6f3f-4b38-b708-2ad312fc3af5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-k46fw\" (UID: \"091b9d41-6f3f-4b38-b708-2ad312fc3af5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw"
Nov 03 20:01:37.281917 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.281881    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzd6q\" (UniqueName: \"kubernetes.io/projected/2b03c6cd-cfd6-4234-820d-d42854548f2d-kube-api-access-nzd6q\") pod \"node-exporter-8b6ng\" (UID: \"2b03c6cd-cfd6-4234-820d-d42854548f2d\") " pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.282325 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.282305    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6ckv\" (UniqueName: \"kubernetes.io/projected/091b9d41-6f3f-4b38-b708-2ad312fc3af5-kube-api-access-w6ckv\") pod \"kube-state-metrics-5dc8d7d89b-k46fw\" (UID: \"091b9d41-6f3f-4b38-b708-2ad312fc3af5\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw"
Nov 03 20:01:37.315160 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.315128    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qsd4s"
Nov 03 20:01:37.359975 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.359941    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw"
Nov 03 20:01:37.389202 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.388950    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8b6ng"
Nov 03 20:01:37.597965 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.597884    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-qsd4s"]
Nov 03 20:01:37.600561 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:37.600533    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3971ef1f_1497_4eb4_b3a0_da2c5a75b619.slice/crio-c5a984b7c3777cd78f6246d6c793f301cbf6d8db98bbc5247e450712c26929d7 WatchSource:0}: Error finding container c5a984b7c3777cd78f6246d6c793f301cbf6d8db98bbc5247e450712c26929d7: Status 404 returned error can't find the container with id c5a984b7c3777cd78f6246d6c793f301cbf6d8db98bbc5247e450712c26929d7
Nov 03 20:01:37.642329 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:37.642305    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw"]
Nov 03 20:01:37.645124 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:37.645095    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod091b9d41_6f3f_4b38_b708_2ad312fc3af5.slice/crio-1555a6a494c3a165429adcaa0aa6aca9cde8c5d136084e71acd8d83f1d27b638 WatchSource:0}: Error finding container 1555a6a494c3a165429adcaa0aa6aca9cde8c5d136084e71acd8d83f1d27b638: Status 404 returned error can't find the container with id 1555a6a494c3a165429adcaa0aa6aca9cde8c5d136084e71acd8d83f1d27b638
Nov 03 20:01:38.125904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.125801    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sx9rt_126ae513-97e3-4ef0-b415-5ec03ea09960/serve-healthcheck-canary/0.log"
Nov 03 20:01:38.153068 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.153033    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw" event={"ID":"091b9d41-6f3f-4b38-b708-2ad312fc3af5","Type":"ContainerStarted","Data":"1555a6a494c3a165429adcaa0aa6aca9cde8c5d136084e71acd8d83f1d27b638"}
Nov 03 20:01:38.154392 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.154366    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8b6ng" event={"ID":"2b03c6cd-cfd6-4234-820d-d42854548f2d","Type":"ContainerStarted","Data":"dc128452293417117ec30a0e8d410e49d508d46c4ad1ba85049698bf08217374"}
Nov 03 20:01:38.156184 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.156160    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qsd4s" event={"ID":"3971ef1f-1497-4eb4-b3a0-da2c5a75b619","Type":"ContainerStarted","Data":"86c0dcfa936c6b3300dc2b82cd7714949b26f2deacfa2faa7dcff1a36ba73e2b"}
Nov 03 20:01:38.156279 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.156203    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qsd4s" event={"ID":"3971ef1f-1497-4eb4-b3a0-da2c5a75b619","Type":"ContainerStarted","Data":"aa45da4e1318075b926a902678db4ef79b3295e8f8a95df7457255d2c206a407"}
Nov 03 20:01:38.156279 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.156220    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qsd4s" event={"ID":"3971ef1f-1497-4eb4-b3a0-da2c5a75b619","Type":"ContainerStarted","Data":"c5a984b7c3777cd78f6246d6c793f301cbf6d8db98bbc5247e450712c26929d7"}
Nov 03 20:01:38.485513 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.484330    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f95dbfdd8-5bmfg"]
Nov 03 20:01:38.486659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.486636    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:01:38.517679 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.517652    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 20:01:38.517851 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.517759    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 20:01:38.517851 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.517807    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 20:01:38.518926 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.518635    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 20:01:38.518926 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.518776    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 20:01:38.518926 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.518819    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-n7ssn\""
Nov 03 20:01:38.518926 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.518889    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 20:01:38.519317 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.519266    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 20:01:38.523692 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.523622    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f95dbfdd8-5bmfg"]
Nov 03 20:01:38.585882 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.585673    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e1f17ab1-ebec-43bb-a354-b1ced1d02478-console-config\") pod \"console-6f95dbfdd8-5bmfg\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") " pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:01:38.585882 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.585802    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1f17ab1-ebec-43bb-a354-b1ced1d02478-console-oauth-config\") pod \"console-6f95dbfdd8-5bmfg\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") " pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:01:38.585882 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.585837    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1f17ab1-ebec-43bb-a354-b1ced1d02478-oauth-serving-cert\") pod \"console-6f95dbfdd8-5bmfg\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") " pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:01:38.586187 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.585958    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1f17ab1-ebec-43bb-a354-b1ced1d02478-tmp\") pod \"console-6f95dbfdd8-5bmfg\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") " pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:01:38.586187 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.585993    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cq5n\" (UniqueName: \"kubernetes.io/projected/e1f17ab1-ebec-43bb-a354-b1ced1d02478-kube-api-access-4cq5n\") pod \"console-6f95dbfdd8-5bmfg\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") " pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:01:38.586187 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.586030    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f17ab1-ebec-43bb-a354-b1ced1d02478-service-ca\") pod \"console-6f95dbfdd8-5bmfg\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") " pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:01:38.586187 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.586080    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1f17ab1-ebec-43bb-a354-b1ced1d02478-console-serving-cert\") pod \"console-6f95dbfdd8-5bmfg\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") " pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:01:38.686962 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.686914    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1f17ab1-ebec-43bb-a354-b1ced1d02478-tmp\") pod \"console-6f95dbfdd8-5bmfg\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") " pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:01:38.686962 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.686967    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cq5n\" (UniqueName: \"kubernetes.io/projected/e1f17ab1-ebec-43bb-a354-b1ced1d02478-kube-api-access-4cq5n\") pod \"console-6f95dbfdd8-5bmfg\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") " pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:01:38.687495 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.687004    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f17ab1-ebec-43bb-a354-b1ced1d02478-service-ca\") pod \"console-6f95dbfdd8-5bmfg\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") " pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:01:38.687495 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.687048    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1f17ab1-ebec-43bb-a354-b1ced1d02478-console-serving-cert\") pod \"console-6f95dbfdd8-5bmfg\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") " pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:01:38.687495 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.687073    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e1f17ab1-ebec-43bb-a354-b1ced1d02478-console-config\") pod \"console-6f95dbfdd8-5bmfg\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") " pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:01:38.687495 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.687123    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1f17ab1-ebec-43bb-a354-b1ced1d02478-console-oauth-config\") pod \"console-6f95dbfdd8-5bmfg\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") " pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:01:38.687495 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.687148    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1f17ab1-ebec-43bb-a354-b1ced1d02478-oauth-serving-cert\") pod \"console-6f95dbfdd8-5bmfg\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") " pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:01:38.687974 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.687955    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1f17ab1-ebec-43bb-a354-b1ced1d02478-oauth-serving-cert\") pod \"console-6f95dbfdd8-5bmfg\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") " pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:01:38.688175 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.688160    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1f17ab1-ebec-43bb-a354-b1ced1d02478-tmp\") pod \"console-6f95dbfdd8-5bmfg\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") " pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:01:38.688903 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.688882    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f17ab1-ebec-43bb-a354-b1ced1d02478-service-ca\") pod \"console-6f95dbfdd8-5bmfg\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") " pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:01:38.689823 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.689769    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e1f17ab1-ebec-43bb-a354-b1ced1d02478-console-config\") pod \"console-6f95dbfdd8-5bmfg\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") " pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:01:38.697245 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.697220    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1f17ab1-ebec-43bb-a354-b1ced1d02478-console-oauth-config\") pod \"console-6f95dbfdd8-5bmfg\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") " pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:01:38.704405 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.704381    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cq5n\" (UniqueName: \"kubernetes.io/projected/e1f17ab1-ebec-43bb-a354-b1ced1d02478-kube-api-access-4cq5n\") pod \"console-6f95dbfdd8-5bmfg\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") " pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:01:38.704510 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.704415    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1f17ab1-ebec-43bb-a354-b1ced1d02478-console-serving-cert\") pod \"console-6f95dbfdd8-5bmfg\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") " pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:01:38.799365 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:38.799315    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:01:39.080656 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:39.080563    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f95dbfdd8-5bmfg"]
Nov 03 20:01:39.090035 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:39.090004    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f17ab1_ebec_43bb_a354_b1ced1d02478.slice/crio-9983b737c52b9e3b8c946f710acf98d9e0a0e4636186d7e6a8fe018e69e47d2a WatchSource:0}: Error finding container 9983b737c52b9e3b8c946f710acf98d9e0a0e4636186d7e6a8fe018e69e47d2a: Status 404 returned error can't find the container with id 9983b737c52b9e3b8c946f710acf98d9e0a0e4636186d7e6a8fe018e69e47d2a
Nov 03 20:01:39.161150 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:39.161111    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f95dbfdd8-5bmfg" event={"ID":"e1f17ab1-ebec-43bb-a354-b1ced1d02478","Type":"ContainerStarted","Data":"9983b737c52b9e3b8c946f710acf98d9e0a0e4636186d7e6a8fe018e69e47d2a"}
Nov 03 20:01:39.163173 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:39.163139    4421 generic.go:356] "Generic (PLEG): container finished" podID="2b03c6cd-cfd6-4234-820d-d42854548f2d" containerID="22f3ebad091cb6df81eb00748186fb6c6c4394fce5f23fba6660cca789a4f1fc" exitCode=0
Nov 03 20:01:39.163321 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:39.163220    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8b6ng" event={"ID":"2b03c6cd-cfd6-4234-820d-d42854548f2d","Type":"ContainerDied","Data":"22f3ebad091cb6df81eb00748186fb6c6c4394fce5f23fba6660cca789a4f1fc"}
Nov 03 20:01:39.166431 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:39.166397    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qsd4s" event={"ID":"3971ef1f-1497-4eb4-b3a0-da2c5a75b619","Type":"ContainerStarted","Data":"e4d8a2ca79021f9605cdb2862af5cf531e44f13824532d888b15b2932432af9b"}
Nov 03 20:01:40.172455 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:40.172413    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw" event={"ID":"091b9d41-6f3f-4b38-b708-2ad312fc3af5","Type":"ContainerStarted","Data":"1b491de2bc734334a7dee25f953a17ab549aa7760ac4964157aff9b591da09c5"}
Nov 03 20:01:40.172954 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:40.172492    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw" event={"ID":"091b9d41-6f3f-4b38-b708-2ad312fc3af5","Type":"ContainerStarted","Data":"2814f1c1125c654fe1e140ca3b326910914309b0248d9121ef369216ce3fba40"}
Nov 03 20:01:40.172954 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:40.172524    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw" event={"ID":"091b9d41-6f3f-4b38-b708-2ad312fc3af5","Type":"ContainerStarted","Data":"045695a4b79de86655b10403a7bdd3b85c771b3723071ab8d415b768f3033bfb"}
Nov 03 20:01:40.177517 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:40.177478    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8b6ng" event={"ID":"2b03c6cd-cfd6-4234-820d-d42854548f2d","Type":"ContainerStarted","Data":"aa3b13d5f5989e9ca04510bf42ca2904dfe029d1f8ce007944b450e8eb485e48"}
Nov 03 20:01:40.177673 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:40.177541    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8b6ng" event={"ID":"2b03c6cd-cfd6-4234-820d-d42854548f2d","Type":"ContainerStarted","Data":"861a5653e85f612d4c9879837a122ae0b5e6589be23c4bf1dd6aae24ade9ac92"}
Nov 03 20:01:40.200393 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:40.200351    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-k46fw" podStartSLOduration=1.31390954 podStartE2EDuration="3.200339061s" totalImagesPullingTime="1.886429521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:40.198568279 +0000 UTC m=+30.331696744" watchObservedRunningTime="2026-11-03 20:01:40.200339061 +0000 UTC m=+30.333467519"
Nov 03 20:01:40.200836 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:40.200810    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qsd4s" podStartSLOduration=3.019736161 podStartE2EDuration="4.200803643s" totalImagesPullingTime="1.181067482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:39.232932667 +0000 UTC m=+29.366061198" watchObservedRunningTime="2026-11-03 20:01:40.200803643 +0000 UTC m=+30.333932104"
Nov 03 20:01:40.225012 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:40.224221    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8b6ng" podStartSLOduration=1.823679996 podStartE2EDuration="3.22420657s" totalImagesPullingTime="1.31539161s" totalInitContainerRuntime="85.134964ms" isStatefulPod=true podCreationTimestamp="2026-11-03 20:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:40.223338557 +0000 UTC m=+30.356467019" watchObservedRunningTime="2026-11-03 20:01:40.22420657 +0000 UTC m=+30.357335031"
Nov 03 20:01:42.053328 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:42.053296    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h6crz"
Nov 03 20:01:43.110047 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:43.110024    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7679cb5584-qgtd8"
Nov 03 20:01:43.192222 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:43.192178    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f95dbfdd8-5bmfg" event={"ID":"e1f17ab1-ebec-43bb-a354-b1ced1d02478","Type":"ContainerStarted","Data":"8c506aa19334acfea6c77d38e0b5edbc3347250bfdcdd080b2656dde2f6ffae0"}
Nov 03 20:01:43.216361 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:43.216307    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6f95dbfdd8-5bmfg" podStartSLOduration=1.3303732990000001 podStartE2EDuration="5.216288622s" totalImagesPullingTime="3.885915323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:43.215709525 +0000 UTC m=+33.348837988" watchObservedRunningTime="2026-11-03 20:01:43.216288622 +0000 UTC m=+33.349417084"
Nov 03 20:01:47.819130 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:47.819098    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-645fd4fdd-cqskf"]
Nov 03 20:01:47.821119 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:47.821098    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:47.836334 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:47.836308    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 20:01:47.837407 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:47.837387    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-645fd4fdd-cqskf"]
Nov 03 20:01:47.963456 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:47.963419    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-service-ca\") pod \"console-645fd4fdd-cqskf\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") " pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:47.963645 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:47.963469    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-trusted-ca-bundle\") pod \"console-645fd4fdd-cqskf\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") " pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:47.963645 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:47.963489    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjh44\" (UniqueName: \"kubernetes.io/projected/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-kube-api-access-rjh44\") pod \"console-645fd4fdd-cqskf\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") " pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:47.963645 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:47.963617    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-console-serving-cert\") pod \"console-645fd4fdd-cqskf\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") " pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:47.963754 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:47.963644    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-console-config\") pod \"console-645fd4fdd-cqskf\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") " pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:47.963754 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:47.963670    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-oauth-serving-cert\") pod \"console-645fd4fdd-cqskf\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") " pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:47.963820 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:47.963746    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-console-oauth-config\") pod \"console-645fd4fdd-cqskf\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") " pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:47.963820 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:47.963783    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-tmp\") pod \"console-645fd4fdd-cqskf\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") " pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:48.065112 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:48.065079    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-console-serving-cert\") pod \"console-645fd4fdd-cqskf\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") " pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:48.065112 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:48.065113    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-console-config\") pod \"console-645fd4fdd-cqskf\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") " pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:48.065403 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:48.065132    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-oauth-serving-cert\") pod \"console-645fd4fdd-cqskf\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") " pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:48.065403 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:48.065148    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-console-oauth-config\") pod \"console-645fd4fdd-cqskf\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") " pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:48.065403 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:48.065166    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-tmp\") pod \"console-645fd4fdd-cqskf\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") " pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:48.065403 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:48.065316    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-service-ca\") pod \"console-645fd4fdd-cqskf\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") " pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:48.065648 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:48.065409    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-trusted-ca-bundle\") pod \"console-645fd4fdd-cqskf\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") " pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:48.065648 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:48.065437    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjh44\" (UniqueName: \"kubernetes.io/projected/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-kube-api-access-rjh44\") pod \"console-645fd4fdd-cqskf\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") " pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:48.065800 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:48.065781    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-tmp\") pod \"console-645fd4fdd-cqskf\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") " pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:48.066090 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:48.066064    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-oauth-serving-cert\") pod \"console-645fd4fdd-cqskf\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") " pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:48.066173 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:48.066075    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-console-config\") pod \"console-645fd4fdd-cqskf\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") " pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:48.066251 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:48.066234    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-service-ca\") pod \"console-645fd4fdd-cqskf\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") " pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:48.066756 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:48.066732    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-trusted-ca-bundle\") pod \"console-645fd4fdd-cqskf\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") " pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:48.067709 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:48.067683    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-console-oauth-config\") pod \"console-645fd4fdd-cqskf\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") " pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:48.067921 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:48.067905    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-console-serving-cert\") pod \"console-645fd4fdd-cqskf\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") " pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:48.077778 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:48.077730    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjh44\" (UniqueName: \"kubernetes.io/projected/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-kube-api-access-rjh44\") pod \"console-645fd4fdd-cqskf\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") " pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:48.131580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:48.131556    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:48.404003 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:48.403932    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-645fd4fdd-cqskf"]
Nov 03 20:01:48.406944 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:01:48.406917    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b473f4_4f6c_46ef_9117_b56c9f3b6d9c.slice/crio-8c9b78ef4522110a4a09634805c8c4bb8877b79c80ea70da0ad660b730134ac9 WatchSource:0}: Error finding container 8c9b78ef4522110a4a09634805c8c4bb8877b79c80ea70da0ad660b730134ac9: Status 404 returned error can't find the container with id 8c9b78ef4522110a4a09634805c8c4bb8877b79c80ea70da0ad660b730134ac9
Nov 03 20:01:48.799994 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:48.799955    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:01:48.800152 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:48.800026    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:01:48.804683 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:48.804663    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:01:49.213792 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:49.213753    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645fd4fdd-cqskf" event={"ID":"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c","Type":"ContainerStarted","Data":"24759a216028e9ac784220959d8ce49ac829c6a657ebeda80fcbbebb551894fa"}
Nov 03 20:01:49.214245 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:49.213807    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645fd4fdd-cqskf" event={"ID":"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c","Type":"ContainerStarted","Data":"8c9b78ef4522110a4a09634805c8c4bb8877b79c80ea70da0ad660b730134ac9"}
Nov 03 20:01:49.217676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:49.217654    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:01:49.234772 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:49.234729    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-645fd4fdd-cqskf" podStartSLOduration=2.234714879 podStartE2EDuration="2.234714879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:49.234021317 +0000 UTC m=+39.367149780" watchObservedRunningTime="2026-11-03 20:01:49.234714879 +0000 UTC m=+39.367843361"
Nov 03 20:01:49.626255 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:01:49.626149    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:01:58.132566 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:58.132529    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:58.133177 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:58.132617    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:58.137564 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:58.137541    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:58.255855 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:58.255829    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:01:58.307052 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:01:58.307020    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f95dbfdd8-5bmfg"]
Nov 03 20:02:19.626261 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:02:19.626220    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:02:23.327725 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.327682    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6f95dbfdd8-5bmfg" podUID="e1f17ab1-ebec-43bb-a354-b1ced1d02478" containerName="console" containerID="cri-o://8c506aa19334acfea6c77d38e0b5edbc3347250bfdcdd080b2656dde2f6ffae0" gracePeriod=15
Nov 03 20:02:23.565290 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.565270    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:02:23.620676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.620610    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1f17ab1-ebec-43bb-a354-b1ced1d02478-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1f17ab1-ebec-43bb-a354-b1ced1d02478-oauth-serving-cert\") pod \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") "
Nov 03 20:02:23.620676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.620650    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f17ab1-ebec-43bb-a354-b1ced1d02478-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1f17ab1-ebec-43bb-a354-b1ced1d02478-tmp\") pod \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") "
Nov 03 20:02:23.620897 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.620681    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1f17ab1-ebec-43bb-a354-b1ced1d02478-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1f17ab1-ebec-43bb-a354-b1ced1d02478-console-serving-cert\") pod \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") "
Nov 03 20:02:23.620897 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.620707    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1f17ab1-ebec-43bb-a354-b1ced1d02478-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1f17ab1-ebec-43bb-a354-b1ced1d02478-console-oauth-config\") pod \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") "
Nov 03 20:02:23.620897 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.620729    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1f17ab1-ebec-43bb-a354-b1ced1d02478-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f17ab1-ebec-43bb-a354-b1ced1d02478-service-ca\") pod \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") "
Nov 03 20:02:23.620897 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.620809    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1f17ab1-ebec-43bb-a354-b1ced1d02478-console-config\" (UniqueName: \"kubernetes.io/configmap/e1f17ab1-ebec-43bb-a354-b1ced1d02478-console-config\") pod \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") "
Nov 03 20:02:23.620897 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.620848    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1f17ab1-ebec-43bb-a354-b1ced1d02478-kube-api-access-4cq5n\" (UniqueName: \"kubernetes.io/projected/e1f17ab1-ebec-43bb-a354-b1ced1d02478-kube-api-access-4cq5n\") pod \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\" (UID: \"e1f17ab1-ebec-43bb-a354-b1ced1d02478\") "
Nov 03 20:02:23.621201 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.621153    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1f17ab1-ebec-43bb-a354-b1ced1d02478-oauth-serving-cert" pod "e1f17ab1-ebec-43bb-a354-b1ced1d02478" (UID: "e1f17ab1-ebec-43bb-a354-b1ced1d02478"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:23.621201 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.621166    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1f17ab1-ebec-43bb-a354-b1ced1d02478-service-ca" pod "e1f17ab1-ebec-43bb-a354-b1ced1d02478" (UID: "e1f17ab1-ebec-43bb-a354-b1ced1d02478"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:23.621201 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.621169    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f17ab1-ebec-43bb-a354-b1ced1d02478-tmp" pod "e1f17ab1-ebec-43bb-a354-b1ced1d02478" (UID: "e1f17ab1-ebec-43bb-a354-b1ced1d02478"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:23.621657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.621494    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1f17ab1-ebec-43bb-a354-b1ced1d02478-console-config" pod "e1f17ab1-ebec-43bb-a354-b1ced1d02478" (UID: "e1f17ab1-ebec-43bb-a354-b1ced1d02478"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:23.623115 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.623093    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1f17ab1-ebec-43bb-a354-b1ced1d02478-console-serving-cert" pod "e1f17ab1-ebec-43bb-a354-b1ced1d02478" (UID: "e1f17ab1-ebec-43bb-a354-b1ced1d02478"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:02:23.623186 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.623151    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1f17ab1-ebec-43bb-a354-b1ced1d02478-console-oauth-config" pod "e1f17ab1-ebec-43bb-a354-b1ced1d02478" (UID: "e1f17ab1-ebec-43bb-a354-b1ced1d02478"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:02:23.623248 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.623230    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f17ab1-ebec-43bb-a354-b1ced1d02478-kube-api-access-4cq5n" pod "e1f17ab1-ebec-43bb-a354-b1ced1d02478" (UID: "e1f17ab1-ebec-43bb-a354-b1ced1d02478"). InnerVolumeSpecName "kube-api-access-4cq5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:23.679114 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.679084    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-8596678df6-sgrph"]
Nov 03 20:02:23.679346 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.679334    4421 cpu_manager.go:414] "containerMap: removing container" podUID="e1f17ab1-ebec-43bb-a354-b1ced1d02478" containerName="console"
Nov 03 20:02:23.679388 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.679349    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1f17ab1-ebec-43bb-a354-b1ced1d02478" containerName="console"
Nov 03 20:02:23.679431 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.679401    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1f17ab1-ebec-43bb-a354-b1ced1d02478" containerName="console"
Nov 03 20:02:23.681117 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.681103    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:23.693856 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.693833    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8596678df6-sgrph"]
Nov 03 20:02:23.721481 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.721452    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14b0cc21-b4a2-4591-854e-91a62ab296c0-console-serving-cert\") pod \"console-8596678df6-sgrph\" (UID: \"14b0cc21-b4a2-4591-854e-91a62ab296c0\") " pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:23.721631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.721489    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14b0cc21-b4a2-4591-854e-91a62ab296c0-oauth-serving-cert\") pod \"console-8596678df6-sgrph\" (UID: \"14b0cc21-b4a2-4591-854e-91a62ab296c0\") " pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:23.721631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.721513    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14b0cc21-b4a2-4591-854e-91a62ab296c0-console-oauth-config\") pod \"console-8596678df6-sgrph\" (UID: \"14b0cc21-b4a2-4591-854e-91a62ab296c0\") " pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:23.721631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.721623    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14b0cc21-b4a2-4591-854e-91a62ab296c0-console-config\") pod \"console-8596678df6-sgrph\" (UID: \"14b0cc21-b4a2-4591-854e-91a62ab296c0\") " pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:23.721787 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.721681    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8m8m\" (UniqueName: \"kubernetes.io/projected/14b0cc21-b4a2-4591-854e-91a62ab296c0-kube-api-access-d8m8m\") pod \"console-8596678df6-sgrph\" (UID: \"14b0cc21-b4a2-4591-854e-91a62ab296c0\") " pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:23.721787 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.721716    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14b0cc21-b4a2-4591-854e-91a62ab296c0-tmp\") pod \"console-8596678df6-sgrph\" (UID: \"14b0cc21-b4a2-4591-854e-91a62ab296c0\") " pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:23.721787 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.721732    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14b0cc21-b4a2-4591-854e-91a62ab296c0-trusted-ca-bundle\") pod \"console-8596678df6-sgrph\" (UID: \"14b0cc21-b4a2-4591-854e-91a62ab296c0\") " pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:23.721787 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.721758    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14b0cc21-b4a2-4591-854e-91a62ab296c0-service-ca\") pod \"console-8596678df6-sgrph\" (UID: \"14b0cc21-b4a2-4591-854e-91a62ab296c0\") " pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:23.721906 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.721841    4421 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e1f17ab1-ebec-43bb-a354-b1ced1d02478-oauth-serving-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:23.721906 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.721852    4421 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1f17ab1-ebec-43bb-a354-b1ced1d02478-tmp\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:23.721906 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.721861    4421 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e1f17ab1-ebec-43bb-a354-b1ced1d02478-console-serving-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:23.721906 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.721869    4421 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e1f17ab1-ebec-43bb-a354-b1ced1d02478-console-oauth-config\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:23.721906 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.721877    4421 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f17ab1-ebec-43bb-a354-b1ced1d02478-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:23.721906 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.721885    4421 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e1f17ab1-ebec-43bb-a354-b1ced1d02478-console-config\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:23.721906 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.721893    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cq5n\" (UniqueName: \"kubernetes.io/projected/e1f17ab1-ebec-43bb-a354-b1ced1d02478-kube-api-access-4cq5n\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:23.822844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.822813    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14b0cc21-b4a2-4591-854e-91a62ab296c0-oauth-serving-cert\") pod \"console-8596678df6-sgrph\" (UID: \"14b0cc21-b4a2-4591-854e-91a62ab296c0\") " pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:23.822976 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.822850    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14b0cc21-b4a2-4591-854e-91a62ab296c0-console-oauth-config\") pod \"console-8596678df6-sgrph\" (UID: \"14b0cc21-b4a2-4591-854e-91a62ab296c0\") " pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:23.822976 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.822868    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14b0cc21-b4a2-4591-854e-91a62ab296c0-console-config\") pod \"console-8596678df6-sgrph\" (UID: \"14b0cc21-b4a2-4591-854e-91a62ab296c0\") " pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:23.822976 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.822892    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8m8m\" (UniqueName: \"kubernetes.io/projected/14b0cc21-b4a2-4591-854e-91a62ab296c0-kube-api-access-d8m8m\") pod \"console-8596678df6-sgrph\" (UID: \"14b0cc21-b4a2-4591-854e-91a62ab296c0\") " pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:23.822976 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.822941    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14b0cc21-b4a2-4591-854e-91a62ab296c0-tmp\") pod \"console-8596678df6-sgrph\" (UID: \"14b0cc21-b4a2-4591-854e-91a62ab296c0\") " pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:23.822976 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.822967    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14b0cc21-b4a2-4591-854e-91a62ab296c0-trusted-ca-bundle\") pod \"console-8596678df6-sgrph\" (UID: \"14b0cc21-b4a2-4591-854e-91a62ab296c0\") " pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:23.823242 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.822991    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14b0cc21-b4a2-4591-854e-91a62ab296c0-service-ca\") pod \"console-8596678df6-sgrph\" (UID: \"14b0cc21-b4a2-4591-854e-91a62ab296c0\") " pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:23.823242 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.823042    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14b0cc21-b4a2-4591-854e-91a62ab296c0-console-serving-cert\") pod \"console-8596678df6-sgrph\" (UID: \"14b0cc21-b4a2-4591-854e-91a62ab296c0\") " pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:23.823565 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.823515    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/14b0cc21-b4a2-4591-854e-91a62ab296c0-tmp\") pod \"console-8596678df6-sgrph\" (UID: \"14b0cc21-b4a2-4591-854e-91a62ab296c0\") " pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:23.823692 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.823659    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/14b0cc21-b4a2-4591-854e-91a62ab296c0-oauth-serving-cert\") pod \"console-8596678df6-sgrph\" (UID: \"14b0cc21-b4a2-4591-854e-91a62ab296c0\") " pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:23.823756 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.823713    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/14b0cc21-b4a2-4591-854e-91a62ab296c0-console-config\") pod \"console-8596678df6-sgrph\" (UID: \"14b0cc21-b4a2-4591-854e-91a62ab296c0\") " pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:23.823853 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.823835    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/14b0cc21-b4a2-4591-854e-91a62ab296c0-service-ca\") pod \"console-8596678df6-sgrph\" (UID: \"14b0cc21-b4a2-4591-854e-91a62ab296c0\") " pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:23.824335 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.824312    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14b0cc21-b4a2-4591-854e-91a62ab296c0-trusted-ca-bundle\") pod \"console-8596678df6-sgrph\" (UID: \"14b0cc21-b4a2-4591-854e-91a62ab296c0\") " pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:23.825565 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.825551    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/14b0cc21-b4a2-4591-854e-91a62ab296c0-console-oauth-config\") pod \"console-8596678df6-sgrph\" (UID: \"14b0cc21-b4a2-4591-854e-91a62ab296c0\") " pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:23.825735 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.825720    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/14b0cc21-b4a2-4591-854e-91a62ab296c0-console-serving-cert\") pod \"console-8596678df6-sgrph\" (UID: \"14b0cc21-b4a2-4591-854e-91a62ab296c0\") " pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:23.835267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.835245    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8m8m\" (UniqueName: \"kubernetes.io/projected/14b0cc21-b4a2-4591-854e-91a62ab296c0-kube-api-access-d8m8m\") pod \"console-8596678df6-sgrph\" (UID: \"14b0cc21-b4a2-4591-854e-91a62ab296c0\") " pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:23.991269 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:23.991239    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:24.263237 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:24.263160    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8596678df6-sgrph"]
Nov 03 20:02:24.265893 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:02:24.265863    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b0cc21_b4a2_4591_854e_91a62ab296c0.slice/crio-e36db1fad7a4e542963f76c8d2bfb157bf44c9357dd821a45c65656d096b6baf WatchSource:0}: Error finding container e36db1fad7a4e542963f76c8d2bfb157bf44c9357dd821a45c65656d096b6baf: Status 404 returned error can't find the container with id e36db1fad7a4e542963f76c8d2bfb157bf44c9357dd821a45c65656d096b6baf
Nov 03 20:02:24.335677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:24.335649    4421 generic.go:356] "Generic (PLEG): container finished" podID="e1f17ab1-ebec-43bb-a354-b1ced1d02478" containerID="8c506aa19334acfea6c77d38e0b5edbc3347250bfdcdd080b2656dde2f6ffae0" exitCode=0
Nov 03 20:02:24.336074 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:24.335693    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f95dbfdd8-5bmfg"
Nov 03 20:02:24.336074 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:24.335735    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f95dbfdd8-5bmfg" event={"ID":"e1f17ab1-ebec-43bb-a354-b1ced1d02478","Type":"ContainerDied","Data":"8c506aa19334acfea6c77d38e0b5edbc3347250bfdcdd080b2656dde2f6ffae0"}
Nov 03 20:02:24.336074 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:24.335792    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f95dbfdd8-5bmfg" event={"ID":"e1f17ab1-ebec-43bb-a354-b1ced1d02478","Type":"ContainerDied","Data":"9983b737c52b9e3b8c946f710acf98d9e0a0e4636186d7e6a8fe018e69e47d2a"}
Nov 03 20:02:24.336074 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:24.335822    4421 scope.go:122] "RemoveContainer" containerID="8c506aa19334acfea6c77d38e0b5edbc3347250bfdcdd080b2656dde2f6ffae0"
Nov 03 20:02:24.337471 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:24.337447    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8596678df6-sgrph" event={"ID":"14b0cc21-b4a2-4591-854e-91a62ab296c0","Type":"ContainerStarted","Data":"0c003728c89a68d29e3d172aa8fca68517c197a6e70abdd68ae84c09dd3a8d78"}
Nov 03 20:02:24.337575 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:24.337491    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8596678df6-sgrph" event={"ID":"14b0cc21-b4a2-4591-854e-91a62ab296c0","Type":"ContainerStarted","Data":"e36db1fad7a4e542963f76c8d2bfb157bf44c9357dd821a45c65656d096b6baf"}
Nov 03 20:02:24.351681 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:24.351662    4421 scope.go:122] "RemoveContainer" containerID="8c506aa19334acfea6c77d38e0b5edbc3347250bfdcdd080b2656dde2f6ffae0"
Nov 03 20:02:24.351966 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:02:24.351946    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c506aa19334acfea6c77d38e0b5edbc3347250bfdcdd080b2656dde2f6ffae0\": container with ID starting with 8c506aa19334acfea6c77d38e0b5edbc3347250bfdcdd080b2656dde2f6ffae0 not found: ID does not exist" containerID="8c506aa19334acfea6c77d38e0b5edbc3347250bfdcdd080b2656dde2f6ffae0"
Nov 03 20:02:24.352042 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:24.351973    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c506aa19334acfea6c77d38e0b5edbc3347250bfdcdd080b2656dde2f6ffae0"} err="failed to get container status \"8c506aa19334acfea6c77d38e0b5edbc3347250bfdcdd080b2656dde2f6ffae0\": rpc error: code = NotFound desc = could not find container \"8c506aa19334acfea6c77d38e0b5edbc3347250bfdcdd080b2656dde2f6ffae0\": container with ID starting with 8c506aa19334acfea6c77d38e0b5edbc3347250bfdcdd080b2656dde2f6ffae0 not found: ID does not exist"
Nov 03 20:02:24.362452 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:24.362389    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-8596678df6-sgrph" podStartSLOduration=1.362372867 podStartE2EDuration="1.362372867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:24.362308967 +0000 UTC m=+74.495437438" watchObservedRunningTime="2026-11-03 20:02:24.362372867 +0000 UTC m=+74.495501339"
Nov 03 20:02:24.378342 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:24.378313    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f95dbfdd8-5bmfg"]
Nov 03 20:02:24.387983 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:24.384361    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f95dbfdd8-5bmfg"]
Nov 03 20:02:26.012967 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:26.012936    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1f17ab1-ebec-43bb-a354-b1ced1d02478" path="/var/lib/kubelet/pods/e1f17ab1-ebec-43bb-a354-b1ced1d02478/volumes"
Nov 03 20:02:33.991381 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:33.991346    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:33.991876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:33.991409    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:33.996006 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:33.995986    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:34.373737 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:34.373662    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8596678df6-sgrph"
Nov 03 20:02:34.425662 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:34.425628    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-645fd4fdd-cqskf"]
Nov 03 20:02:49.625920 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:02:49.625883    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:02:59.450916 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:59.450874    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-645fd4fdd-cqskf" podUID="f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c" containerName="console" containerID="cri-o://24759a216028e9ac784220959d8ce49ac829c6a657ebeda80fcbbebb551894fa" gracePeriod=15
Nov 03 20:02:59.682820 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:59.682797    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:02:59.779995 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:59.779903    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-console-config\" (UniqueName: \"kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-console-config\") pod \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") "
Nov 03 20:02:59.779995 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:59.779965    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-console-oauth-config\") pod \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") "
Nov 03 20:02:59.780223 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:59.779997    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-console-serving-cert\") pod \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") "
Nov 03 20:02:59.780223 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:59.780108    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-tmp\") pod \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") "
Nov 03 20:02:59.780223 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:59.780149    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-trusted-ca-bundle\") pod \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") "
Nov 03 20:02:59.780223 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:59.780194    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-oauth-serving-cert\") pod \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") "
Nov 03 20:02:59.780439 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:59.780235    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-service-ca\") pod \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") "
Nov 03 20:02:59.780439 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:59.780304    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-kube-api-access-rjh44\" (UniqueName: \"kubernetes.io/projected/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-kube-api-access-rjh44\") pod \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\" (UID: \"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c\") "
Nov 03 20:02:59.780439 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:59.780341    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-console-config" pod "f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c" (UID: "f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:59.780587 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:59.780448    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-tmp" pod "f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c" (UID: "f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:59.780587 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:59.780539    4421 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-console-config\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:59.780587 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:59.780555    4421 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-tmp\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:59.780876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:59.780710    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-oauth-serving-cert" pod "f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c" (UID: "f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:59.780876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:59.780727    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-service-ca" pod "f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c" (UID: "f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:59.780876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:59.780747    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-trusted-ca-bundle" pod "f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c" (UID: "f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:59.782358 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:59.782334    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-console-serving-cert" pod "f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c" (UID: "f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:02:59.782467 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:59.782412    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-console-oauth-config" pod "f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c" (UID: "f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:02:59.782642 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:59.782620    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-kube-api-access-rjh44" pod "f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c" (UID: "f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c"). InnerVolumeSpecName "kube-api-access-rjh44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:59.881790 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:59.881761    4421 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-console-oauth-config\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:59.881790 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:59.881785    4421 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-console-serving-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:59.881790 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:59.881794    4421 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-trusted-ca-bundle\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:59.881990 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:59.881803    4421 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-oauth-serving-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:59.881990 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:59.881813    4421 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:59.881990 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:02:59.881821    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjh44\" (UniqueName: \"kubernetes.io/projected/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c-kube-api-access-rjh44\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:00.458076 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:03:00.458039    4421 generic.go:356] "Generic (PLEG): container finished" podID="f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c" containerID="24759a216028e9ac784220959d8ce49ac829c6a657ebeda80fcbbebb551894fa" exitCode=0
Nov 03 20:03:00.458521 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:03:00.458091    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645fd4fdd-cqskf" event={"ID":"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c","Type":"ContainerDied","Data":"24759a216028e9ac784220959d8ce49ac829c6a657ebeda80fcbbebb551894fa"}
Nov 03 20:03:00.458521 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:03:00.458130    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645fd4fdd-cqskf" event={"ID":"f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c","Type":"ContainerDied","Data":"8c9b78ef4522110a4a09634805c8c4bb8877b79c80ea70da0ad660b730134ac9"}
Nov 03 20:03:00.458521 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:03:00.458145    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645fd4fdd-cqskf"
Nov 03 20:03:00.458521 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:03:00.458154    4421 scope.go:122] "RemoveContainer" containerID="24759a216028e9ac784220959d8ce49ac829c6a657ebeda80fcbbebb551894fa"
Nov 03 20:03:00.465899 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:03:00.465881    4421 scope.go:122] "RemoveContainer" containerID="24759a216028e9ac784220959d8ce49ac829c6a657ebeda80fcbbebb551894fa"
Nov 03 20:03:00.466160 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:03:00.466136    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24759a216028e9ac784220959d8ce49ac829c6a657ebeda80fcbbebb551894fa\": container with ID starting with 24759a216028e9ac784220959d8ce49ac829c6a657ebeda80fcbbebb551894fa not found: ID does not exist" containerID="24759a216028e9ac784220959d8ce49ac829c6a657ebeda80fcbbebb551894fa"
Nov 03 20:03:00.466219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:03:00.466174    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24759a216028e9ac784220959d8ce49ac829c6a657ebeda80fcbbebb551894fa"} err="failed to get container status \"24759a216028e9ac784220959d8ce49ac829c6a657ebeda80fcbbebb551894fa\": rpc error: code = NotFound desc = could not find container \"24759a216028e9ac784220959d8ce49ac829c6a657ebeda80fcbbebb551894fa\": container with ID starting with 24759a216028e9ac784220959d8ce49ac829c6a657ebeda80fcbbebb551894fa not found: ID does not exist"
Nov 03 20:03:00.475808 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:03:00.475785    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-645fd4fdd-cqskf"]
Nov 03 20:03:00.479737 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:03:00.479717    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-645fd4fdd-cqskf"]
Nov 03 20:03:02.012474 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:03:02.012435    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c" path="/var/lib/kubelet/pods/f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c/volumes"
Nov 03 20:03:19.626478 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:03:19.626396    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:03:49.626699 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:03:49.626663    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:04:19.627002 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:04:19.626966    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:04:49.626359 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:04:49.626273    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:05:19.626935 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:05:19.626890    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:05:49.626282 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:05:49.626246    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:05:50.416948 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.416917    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7"]
Nov 03 20:05:50.417181 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.417169    4421 cpu_manager.go:414] "containerMap: removing container" podUID="f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c" containerName="console"
Nov 03 20:05:50.417226 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.417182    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c" containerName="console"
Nov 03 20:05:50.417259 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.417231    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b473f4-4f6c-46ef-9117-b56c9f3b6d9c" containerName="console"
Nov 03 20:05:50.421388 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.421369    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7"
Nov 03 20:05:50.433428 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.433407    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 20:05:50.433428 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.433407    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 20:05:50.433576 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.433416    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-zj5gs\""
Nov 03 20:05:50.442549 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.442527    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7"]
Nov 03 20:05:50.476038 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.476011    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkrqw\" (UniqueName: \"kubernetes.io/projected/9497e06f-29f6-444b-a7aa-cab2ac7c8db9-kube-api-access-qkrqw\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7\" (UID: \"9497e06f-29f6-444b-a7aa-cab2ac7c8db9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7"
Nov 03 20:05:50.476213 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.476071    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9497e06f-29f6-444b-a7aa-cab2ac7c8db9-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7\" (UID: \"9497e06f-29f6-444b-a7aa-cab2ac7c8db9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7"
Nov 03 20:05:50.476213 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.476195    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9497e06f-29f6-444b-a7aa-cab2ac7c8db9-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7\" (UID: \"9497e06f-29f6-444b-a7aa-cab2ac7c8db9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7"
Nov 03 20:05:50.576736 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.576699    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkrqw\" (UniqueName: \"kubernetes.io/projected/9497e06f-29f6-444b-a7aa-cab2ac7c8db9-kube-api-access-qkrqw\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7\" (UID: \"9497e06f-29f6-444b-a7aa-cab2ac7c8db9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7"
Nov 03 20:05:50.576901 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.576742    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9497e06f-29f6-444b-a7aa-cab2ac7c8db9-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7\" (UID: \"9497e06f-29f6-444b-a7aa-cab2ac7c8db9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7"
Nov 03 20:05:50.576966 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.576892    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9497e06f-29f6-444b-a7aa-cab2ac7c8db9-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7\" (UID: \"9497e06f-29f6-444b-a7aa-cab2ac7c8db9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7"
Nov 03 20:05:50.577228 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.577207    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9497e06f-29f6-444b-a7aa-cab2ac7c8db9-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7\" (UID: \"9497e06f-29f6-444b-a7aa-cab2ac7c8db9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7"
Nov 03 20:05:50.577287 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.577227    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9497e06f-29f6-444b-a7aa-cab2ac7c8db9-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7\" (UID: \"9497e06f-29f6-444b-a7aa-cab2ac7c8db9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7"
Nov 03 20:05:50.594917 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.594892    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkrqw\" (UniqueName: \"kubernetes.io/projected/9497e06f-29f6-444b-a7aa-cab2ac7c8db9-kube-api-access-qkrqw\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7\" (UID: \"9497e06f-29f6-444b-a7aa-cab2ac7c8db9\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7"
Nov 03 20:05:50.739816 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.739784    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7"
Nov 03 20:05:50.896503 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.896468    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"]
Nov 03 20:05:50.899799 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.899782    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"
Nov 03 20:05:50.905072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.904982    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 20:05:50.905072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.905034    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 20:05:50.905072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.905046    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 20:05:50.905072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.905064    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 20:05:50.906322 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.906299    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 20:05:50.906407 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.906356    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kbftm\""
Nov 03 20:05:50.906407 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.906361    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 20:05:50.915378 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.915352    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"]
Nov 03 20:05:50.980090 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.980065    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-kw7bs\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"
Nov 03 20:05:50.980198 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.980093    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmwc6\" (UniqueName: \"kubernetes.io/projected/12e64430-b367-4411-b366-d7c0f6e19eaf-kube-api-access-qmwc6\") pod \"istiod-openshift-gateway-7876bfcb74-kw7bs\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"
Nov 03 20:05:50.980198 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.980114    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-kw7bs\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"
Nov 03 20:05:50.980198 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.980188    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-kw7bs\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"
Nov 03 20:05:50.980298 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.980269    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/12e64430-b367-4411-b366-d7c0f6e19eaf-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-kw7bs\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"
Nov 03 20:05:50.980331 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.980300    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e64430-b367-4411-b366-d7c0f6e19eaf-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-kw7bs\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"
Nov 03 20:05:50.980331 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:50.980325    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-kw7bs\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"
Nov 03 20:05:51.022955 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:51.022935    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7"]
Nov 03 20:05:51.025072 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:05:51.025047    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9497e06f_29f6_444b_a7aa_cab2ac7c8db9.slice/crio-485e839fe1d22f97173150ae3df1970632485cdec0e61845e88790587e8fa2ed WatchSource:0}: Error finding container 485e839fe1d22f97173150ae3df1970632485cdec0e61845e88790587e8fa2ed: Status 404 returned error can't find the container with id 485e839fe1d22f97173150ae3df1970632485cdec0e61845e88790587e8fa2ed
Nov 03 20:05:51.081206 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:51.081179    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/12e64430-b367-4411-b366-d7c0f6e19eaf-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-kw7bs\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"
Nov 03 20:05:51.081368 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:51.081217    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e64430-b367-4411-b366-d7c0f6e19eaf-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-kw7bs\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"
Nov 03 20:05:51.081368 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:51.081247    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-kw7bs\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"
Nov 03 20:05:51.081368 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:51.081295    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-kw7bs\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"
Nov 03 20:05:51.081368 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:51.081323    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmwc6\" (UniqueName: \"kubernetes.io/projected/12e64430-b367-4411-b366-d7c0f6e19eaf-kube-api-access-qmwc6\") pod \"istiod-openshift-gateway-7876bfcb74-kw7bs\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"
Nov 03 20:05:51.081368 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:51.081351    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-kw7bs\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"
Nov 03 20:05:51.081659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:51.081635    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-kw7bs\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"
Nov 03 20:05:51.082032 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:51.082008    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-kw7bs\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"
Nov 03 20:05:51.083743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:51.083715    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e64430-b367-4411-b366-d7c0f6e19eaf-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-kw7bs\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"
Nov 03 20:05:51.083909 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:51.083890    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-kw7bs\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"
Nov 03 20:05:51.084072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:51.084059    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/12e64430-b367-4411-b366-d7c0f6e19eaf-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-kw7bs\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"
Nov 03 20:05:51.084119 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:51.084104    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-kw7bs\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"
Nov 03 20:05:51.093641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:51.093622    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-kw7bs\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"
Nov 03 20:05:51.093986 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:51.093968    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmwc6\" (UniqueName: \"kubernetes.io/projected/12e64430-b367-4411-b366-d7c0f6e19eaf-kube-api-access-qmwc6\") pod \"istiod-openshift-gateway-7876bfcb74-kw7bs\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"
Nov 03 20:05:51.211396 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:51.211365    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"
Nov 03 20:05:51.488940 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:51.488917    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"]
Nov 03 20:05:51.491528 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:05:51.491502    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e64430_b367_4411_b366_d7c0f6e19eaf.slice/crio-22bac99d91f7a7aa390e18b91e7a8aebe7edaebd98801a912738e778d7cd3039 WatchSource:0}: Error finding container 22bac99d91f7a7aa390e18b91e7a8aebe7edaebd98801a912738e778d7cd3039: Status 404 returned error can't find the container with id 22bac99d91f7a7aa390e18b91e7a8aebe7edaebd98801a912738e778d7cd3039
Nov 03 20:05:51.995663 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:51.995618    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs" event={"ID":"12e64430-b367-4411-b366-d7c0f6e19eaf","Type":"ContainerStarted","Data":"22bac99d91f7a7aa390e18b91e7a8aebe7edaebd98801a912738e778d7cd3039"}
Nov 03 20:05:51.997086 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:51.997061    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7" event={"ID":"9497e06f-29f6-444b-a7aa-cab2ac7c8db9","Type":"ContainerStarted","Data":"485e839fe1d22f97173150ae3df1970632485cdec0e61845e88790587e8fa2ed"}
Nov 03 20:05:57.156789 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:57.156759    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:05:57.157071 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:57.156844    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:05:58.022472 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:58.022422    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs" event={"ID":"12e64430-b367-4411-b366-d7c0f6e19eaf","Type":"ContainerStarted","Data":"b33046188ecb1076b70ef0188b25ba7a61a81a9b2b1c173bea1372e1a023991c"}
Nov 03 20:05:58.022654 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:58.022609    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"
Nov 03 20:05:58.024292 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:58.024263    4421 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-kw7bs container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:05:58.024395 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:58.024292    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7" event={"ID":"9497e06f-29f6-444b-a7aa-cab2ac7c8db9","Type":"ContainerDied","Data":"1b8b7fd4229f2c6b375d0fb400a68a13600aba25d8f0cbbc61b49cc55d0c2a40"}
Nov 03 20:05:58.024395 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:58.024269    4421 generic.go:356] "Generic (PLEG): container finished" podID="9497e06f-29f6-444b-a7aa-cab2ac7c8db9" containerID="1b8b7fd4229f2c6b375d0fb400a68a13600aba25d8f0cbbc61b49cc55d0c2a40" exitCode=0
Nov 03 20:05:58.024474 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:58.024320    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs" podUID="12e64430-b367-4411-b366-d7c0f6e19eaf" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:05:58.048141 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:58.048096    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs" podStartSLOduration=2.385613819 podStartE2EDuration="8.048085707s" totalImagesPullingTime="5.662471888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:58.046399406 +0000 UTC m=+288.179527882" watchObservedRunningTime="2026-11-03 20:05:58.048085707 +0000 UTC m=+288.181214169"
Nov 03 20:05:59.028150 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:05:59.028121    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"
Nov 03 20:06:04.044090 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:04.044001    4421 generic.go:356] "Generic (PLEG): container finished" podID="9497e06f-29f6-444b-a7aa-cab2ac7c8db9" containerID="74beada470d5136e9a9b311be74e287cbf982ed8c1b8a0d285247422708e3120" exitCode=0
Nov 03 20:06:04.044090 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:04.044039    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7" event={"ID":"9497e06f-29f6-444b-a7aa-cab2ac7c8db9","Type":"ContainerDied","Data":"74beada470d5136e9a9b311be74e287cbf982ed8c1b8a0d285247422708e3120"}
Nov 03 20:06:11.069859 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:11.069819    4421 generic.go:356] "Generic (PLEG): container finished" podID="9497e06f-29f6-444b-a7aa-cab2ac7c8db9" containerID="664dce018e788dfdd1080727f0a00133f79d85341e5088dd03512c85caa02ef5" exitCode=0
Nov 03 20:06:11.070329 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:11.069900    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7" event={"ID":"9497e06f-29f6-444b-a7aa-cab2ac7c8db9","Type":"ContainerDied","Data":"664dce018e788dfdd1080727f0a00133f79d85341e5088dd03512c85caa02ef5"}
Nov 03 20:06:12.196728 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:12.196699    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7"
Nov 03 20:06:12.350844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:12.350769    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9497e06f-29f6-444b-a7aa-cab2ac7c8db9-kube-api-access-qkrqw\" (UniqueName: \"kubernetes.io/projected/9497e06f-29f6-444b-a7aa-cab2ac7c8db9-kube-api-access-qkrqw\") pod \"9497e06f-29f6-444b-a7aa-cab2ac7c8db9\" (UID: \"9497e06f-29f6-444b-a7aa-cab2ac7c8db9\") "
Nov 03 20:06:12.350844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:12.350814    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9497e06f-29f6-444b-a7aa-cab2ac7c8db9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9497e06f-29f6-444b-a7aa-cab2ac7c8db9-bundle\") pod \"9497e06f-29f6-444b-a7aa-cab2ac7c8db9\" (UID: \"9497e06f-29f6-444b-a7aa-cab2ac7c8db9\") "
Nov 03 20:06:12.351058 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:12.350868    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9497e06f-29f6-444b-a7aa-cab2ac7c8db9-util\" (UniqueName: \"kubernetes.io/empty-dir/9497e06f-29f6-444b-a7aa-cab2ac7c8db9-util\") pod \"9497e06f-29f6-444b-a7aa-cab2ac7c8db9\" (UID: \"9497e06f-29f6-444b-a7aa-cab2ac7c8db9\") "
Nov 03 20:06:12.351511 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:12.351483    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9497e06f-29f6-444b-a7aa-cab2ac7c8db9-bundle" pod "9497e06f-29f6-444b-a7aa-cab2ac7c8db9" (UID: "9497e06f-29f6-444b-a7aa-cab2ac7c8db9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:12.353099 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:12.353079    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9497e06f-29f6-444b-a7aa-cab2ac7c8db9-kube-api-access-qkrqw" pod "9497e06f-29f6-444b-a7aa-cab2ac7c8db9" (UID: "9497e06f-29f6-444b-a7aa-cab2ac7c8db9"). InnerVolumeSpecName "kube-api-access-qkrqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:12.356228 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:12.356202    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9497e06f-29f6-444b-a7aa-cab2ac7c8db9-util" pod "9497e06f-29f6-444b-a7aa-cab2ac7c8db9" (UID: "9497e06f-29f6-444b-a7aa-cab2ac7c8db9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:12.451716 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:12.451687    4421 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9497e06f-29f6-444b-a7aa-cab2ac7c8db9-util\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:12.451716 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:12.451714    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkrqw\" (UniqueName: \"kubernetes.io/projected/9497e06f-29f6-444b-a7aa-cab2ac7c8db9-kube-api-access-qkrqw\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:12.451885 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:12.451725    4421 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9497e06f-29f6-444b-a7aa-cab2ac7c8db9-bundle\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:13.077810 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:13.077770    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7" event={"ID":"9497e06f-29f6-444b-a7aa-cab2ac7c8db9","Type":"ContainerDied","Data":"485e839fe1d22f97173150ae3df1970632485cdec0e61845e88790587e8fa2ed"}
Nov 03 20:06:13.077810 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:13.077797    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cslmx7"
Nov 03 20:06:13.077810 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:13.077817    4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="485e839fe1d22f97173150ae3df1970632485cdec0e61845e88790587e8fa2ed"
Nov 03 20:06:19.626515 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:06:19.626434    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:06:48.065918 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.065882    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n"]
Nov 03 20:06:48.066512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.066221    4421 cpu_manager.go:414] "containerMap: removing container" podUID="9497e06f-29f6-444b-a7aa-cab2ac7c8db9" containerName="util"
Nov 03 20:06:48.066512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.066240    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9497e06f-29f6-444b-a7aa-cab2ac7c8db9" containerName="util"
Nov 03 20:06:48.066512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.066264    4421 cpu_manager.go:414] "containerMap: removing container" podUID="9497e06f-29f6-444b-a7aa-cab2ac7c8db9" containerName="extract"
Nov 03 20:06:48.066512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.066271    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9497e06f-29f6-444b-a7aa-cab2ac7c8db9" containerName="extract"
Nov 03 20:06:48.066512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.066302    4421 cpu_manager.go:414] "containerMap: removing container" podUID="9497e06f-29f6-444b-a7aa-cab2ac7c8db9" containerName="pull"
Nov 03 20:06:48.066512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.066310    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9497e06f-29f6-444b-a7aa-cab2ac7c8db9" containerName="pull"
Nov 03 20:06:48.066512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.066376    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="9497e06f-29f6-444b-a7aa-cab2ac7c8db9" containerName="extract"
Nov 03 20:06:48.068565 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.068546    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n"
Nov 03 20:06:48.106169 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.106139    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n"]
Nov 03 20:06:48.112056 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.112027    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/116d09e9-64bc-4999-9361-8a045fae521b-tmp\") pod \"opendatahub-operator-controller-manager-8865cc5f6-fgw8n\" (UID: \"116d09e9-64bc-4999-9361-8a045fae521b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n"
Nov 03 20:06:48.112172 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.112078    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/116d09e9-64bc-4999-9361-8a045fae521b-manifests\") pod \"opendatahub-operator-controller-manager-8865cc5f6-fgw8n\" (UID: \"116d09e9-64bc-4999-9361-8a045fae521b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n"
Nov 03 20:06:48.112220 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.112184    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjt28\" (UniqueName: \"kubernetes.io/projected/116d09e9-64bc-4999-9361-8a045fae521b-kube-api-access-gjt28\") pod \"opendatahub-operator-controller-manager-8865cc5f6-fgw8n\" (UID: \"116d09e9-64bc-4999-9361-8a045fae521b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n"
Nov 03 20:06:48.112255 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.112229    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/116d09e9-64bc-4999-9361-8a045fae521b-webhook-cert\") pod \"opendatahub-operator-controller-manager-8865cc5f6-fgw8n\" (UID: \"116d09e9-64bc-4999-9361-8a045fae521b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n"
Nov 03 20:06:48.112287 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.112258    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/116d09e9-64bc-4999-9361-8a045fae521b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8865cc5f6-fgw8n\" (UID: \"116d09e9-64bc-4999-9361-8a045fae521b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n"
Nov 03 20:06:48.112358 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.112338    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/116d09e9-64bc-4999-9361-8a045fae521b-metrics-cert\") pod \"opendatahub-operator-controller-manager-8865cc5f6-fgw8n\" (UID: \"116d09e9-64bc-4999-9361-8a045fae521b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n"
Nov 03 20:06:48.119659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.119638    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 20:06:48.119659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.119651    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 20:06:48.141065 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.141037    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 20:06:48.141065 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.141047    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 20:06:48.141065 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.141033    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 20:06:48.141262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.141109    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-m6wx9\""
Nov 03 20:06:48.213724 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.213688    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/116d09e9-64bc-4999-9361-8a045fae521b-tmp\") pod \"opendatahub-operator-controller-manager-8865cc5f6-fgw8n\" (UID: \"116d09e9-64bc-4999-9361-8a045fae521b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n"
Nov 03 20:06:48.213900 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.213732    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/116d09e9-64bc-4999-9361-8a045fae521b-manifests\") pod \"opendatahub-operator-controller-manager-8865cc5f6-fgw8n\" (UID: \"116d09e9-64bc-4999-9361-8a045fae521b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n"
Nov 03 20:06:48.213900 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.213754    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjt28\" (UniqueName: \"kubernetes.io/projected/116d09e9-64bc-4999-9361-8a045fae521b-kube-api-access-gjt28\") pod \"opendatahub-operator-controller-manager-8865cc5f6-fgw8n\" (UID: \"116d09e9-64bc-4999-9361-8a045fae521b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n"
Nov 03 20:06:48.213900 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.213785    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/116d09e9-64bc-4999-9361-8a045fae521b-webhook-cert\") pod \"opendatahub-operator-controller-manager-8865cc5f6-fgw8n\" (UID: \"116d09e9-64bc-4999-9361-8a045fae521b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n"
Nov 03 20:06:48.213900 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.213803    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/116d09e9-64bc-4999-9361-8a045fae521b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8865cc5f6-fgw8n\" (UID: \"116d09e9-64bc-4999-9361-8a045fae521b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n"
Nov 03 20:06:48.214118 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.213959    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/116d09e9-64bc-4999-9361-8a045fae521b-metrics-cert\") pod \"opendatahub-operator-controller-manager-8865cc5f6-fgw8n\" (UID: \"116d09e9-64bc-4999-9361-8a045fae521b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n"
Nov 03 20:06:48.214262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.214240    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/116d09e9-64bc-4999-9361-8a045fae521b-manifests\") pod \"opendatahub-operator-controller-manager-8865cc5f6-fgw8n\" (UID: \"116d09e9-64bc-4999-9361-8a045fae521b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n"
Nov 03 20:06:48.216209 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.216187    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/116d09e9-64bc-4999-9361-8a045fae521b-tmp\") pod \"opendatahub-operator-controller-manager-8865cc5f6-fgw8n\" (UID: \"116d09e9-64bc-4999-9361-8a045fae521b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n"
Nov 03 20:06:48.216415 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.216397    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/116d09e9-64bc-4999-9361-8a045fae521b-webhook-cert\") pod \"opendatahub-operator-controller-manager-8865cc5f6-fgw8n\" (UID: \"116d09e9-64bc-4999-9361-8a045fae521b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n"
Nov 03 20:06:48.216832 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.216813    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/116d09e9-64bc-4999-9361-8a045fae521b-metrics-cert\") pod \"opendatahub-operator-controller-manager-8865cc5f6-fgw8n\" (UID: \"116d09e9-64bc-4999-9361-8a045fae521b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n"
Nov 03 20:06:48.216939 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.216886    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/116d09e9-64bc-4999-9361-8a045fae521b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8865cc5f6-fgw8n\" (UID: \"116d09e9-64bc-4999-9361-8a045fae521b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n"
Nov 03 20:06:48.226498 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.226479    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjt28\" (UniqueName: \"kubernetes.io/projected/116d09e9-64bc-4999-9361-8a045fae521b-kube-api-access-gjt28\") pod \"opendatahub-operator-controller-manager-8865cc5f6-fgw8n\" (UID: \"116d09e9-64bc-4999-9361-8a045fae521b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n"
Nov 03 20:06:48.381576 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.381490    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n"
Nov 03 20:06:48.679829 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.679738    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n"]
Nov 03 20:06:48.689176 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:06:48.682228    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116d09e9_64bc_4999_9361_8a045fae521b.slice/crio-cf18bae1a70fc9e3c104ef5c5a89b197fd0d0b7d808bed4b5b06fa239b25b69e WatchSource:0}: Error finding container cf18bae1a70fc9e3c104ef5c5a89b197fd0d0b7d808bed4b5b06fa239b25b69e: Status 404 returned error can't find the container with id cf18bae1a70fc9e3c104ef5c5a89b197fd0d0b7d808bed4b5b06fa239b25b69e
Nov 03 20:06:48.689176 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:48.684441    4421 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:06:49.194304 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:49.194273    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n" event={"ID":"116d09e9-64bc-4999-9361-8a045fae521b","Type":"ContainerStarted","Data":"cf18bae1a70fc9e3c104ef5c5a89b197fd0d0b7d808bed4b5b06fa239b25b69e"}
Nov 03 20:06:49.633575 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:06:49.633462    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:06:53.211491 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:53.211464    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n" event={"ID":"116d09e9-64bc-4999-9361-8a045fae521b","Type":"ContainerStarted","Data":"6c8f9181670e69523caf29656c53e65de1abe729d3c7793a9e8ec984fb1c59e6"}
Nov 03 20:06:54.216177 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:54.216139    4421 generic.go:356] "Generic (PLEG): container finished" podID="116d09e9-64bc-4999-9361-8a045fae521b" containerID="6c8f9181670e69523caf29656c53e65de1abe729d3c7793a9e8ec984fb1c59e6" exitCode=0
Nov 03 20:06:54.216559 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:54.216224    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n" event={"ID":"116d09e9-64bc-4999-9361-8a045fae521b","Type":"ContainerDied","Data":"6c8f9181670e69523caf29656c53e65de1abe729d3c7793a9e8ec984fb1c59e6"}
Nov 03 20:06:55.221610 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:55.221565    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n" event={"ID":"116d09e9-64bc-4999-9361-8a045fae521b","Type":"ContainerStarted","Data":"4cc16a15a5ebe9bc01192077f6aca541d9ac448ca1b43d302aa75db4e7269054"}
Nov 03 20:06:55.222000 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:55.221691    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n"
Nov 03 20:06:55.267861 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:06:55.267819    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n" podStartSLOduration=2.337764799 podStartE2EDuration="7.267795038s" totalImagesPullingTime="4.458777957s" totalInitContainerRuntime="471.252282ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:55.265876689 +0000 UTC m=+345.399005151" watchObservedRunningTime="2026-11-03 20:06:55.267795038 +0000 UTC m=+345.400923500"
Nov 03 20:07:06.227143 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:06.227115    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-fgw8n"
Nov 03 20:07:19.627519 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:07:19.627482    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:07:22.767697 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:22.767661    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-594f764c95-z8fjv"]
Nov 03 20:07:22.771105 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:22.771084    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-594f764c95-z8fjv"
Nov 03 20:07:22.780148 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:22.780121    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 20:07:22.780376 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:22.780251    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-gkjjr\""
Nov 03 20:07:22.780376 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:22.780337    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 20:07:22.780915 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:22.780893    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-594f764c95-z8fjv"]
Nov 03 20:07:22.896775 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:22.896741    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5520324-5b0c-4417-8221-a2cf1c08b881-tls-certs\") pod \"kube-auth-proxy-594f764c95-z8fjv\" (UID: \"b5520324-5b0c-4417-8221-a2cf1c08b881\") " pod="openshift-ingress/kube-auth-proxy-594f764c95-z8fjv"
Nov 03 20:07:22.896949 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:22.896801    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5520324-5b0c-4417-8221-a2cf1c08b881-tmp\") pod \"kube-auth-proxy-594f764c95-z8fjv\" (UID: \"b5520324-5b0c-4417-8221-a2cf1c08b881\") " pod="openshift-ingress/kube-auth-proxy-594f764c95-z8fjv"
Nov 03 20:07:22.896949 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:22.896921    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps8dg\" (UniqueName: \"kubernetes.io/projected/b5520324-5b0c-4417-8221-a2cf1c08b881-kube-api-access-ps8dg\") pod \"kube-auth-proxy-594f764c95-z8fjv\" (UID: \"b5520324-5b0c-4417-8221-a2cf1c08b881\") " pod="openshift-ingress/kube-auth-proxy-594f764c95-z8fjv"
Nov 03 20:07:22.997900 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:22.997869    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5520324-5b0c-4417-8221-a2cf1c08b881-tmp\") pod \"kube-auth-proxy-594f764c95-z8fjv\" (UID: \"b5520324-5b0c-4417-8221-a2cf1c08b881\") " pod="openshift-ingress/kube-auth-proxy-594f764c95-z8fjv"
Nov 03 20:07:22.998086 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:22.997920    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps8dg\" (UniqueName: \"kubernetes.io/projected/b5520324-5b0c-4417-8221-a2cf1c08b881-kube-api-access-ps8dg\") pod \"kube-auth-proxy-594f764c95-z8fjv\" (UID: \"b5520324-5b0c-4417-8221-a2cf1c08b881\") " pod="openshift-ingress/kube-auth-proxy-594f764c95-z8fjv"
Nov 03 20:07:22.998086 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:22.997940    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5520324-5b0c-4417-8221-a2cf1c08b881-tls-certs\") pod \"kube-auth-proxy-594f764c95-z8fjv\" (UID: \"b5520324-5b0c-4417-8221-a2cf1c08b881\") " pod="openshift-ingress/kube-auth-proxy-594f764c95-z8fjv"
Nov 03 20:07:23.000220 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:23.000193    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5520324-5b0c-4417-8221-a2cf1c08b881-tmp\") pod \"kube-auth-proxy-594f764c95-z8fjv\" (UID: \"b5520324-5b0c-4417-8221-a2cf1c08b881\") " pod="openshift-ingress/kube-auth-proxy-594f764c95-z8fjv"
Nov 03 20:07:23.000718 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:23.000680    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5520324-5b0c-4417-8221-a2cf1c08b881-tls-certs\") pod \"kube-auth-proxy-594f764c95-z8fjv\" (UID: \"b5520324-5b0c-4417-8221-a2cf1c08b881\") " pod="openshift-ingress/kube-auth-proxy-594f764c95-z8fjv"
Nov 03 20:07:23.019227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:23.019161    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps8dg\" (UniqueName: \"kubernetes.io/projected/b5520324-5b0c-4417-8221-a2cf1c08b881-kube-api-access-ps8dg\") pod \"kube-auth-proxy-594f764c95-z8fjv\" (UID: \"b5520324-5b0c-4417-8221-a2cf1c08b881\") " pod="openshift-ingress/kube-auth-proxy-594f764c95-z8fjv"
Nov 03 20:07:23.085703 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:23.085670    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-594f764c95-z8fjv"
Nov 03 20:07:23.361127 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:23.361057    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-594f764c95-z8fjv"]
Nov 03 20:07:23.364539 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:07:23.364511    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5520324_5b0c_4417_8221_a2cf1c08b881.slice/crio-2c22210631cd937734f49f412c11bb50ca344eb55deb78b0f28454fe8508d22e WatchSource:0}: Error finding container 2c22210631cd937734f49f412c11bb50ca344eb55deb78b0f28454fe8508d22e: Status 404 returned error can't find the container with id 2c22210631cd937734f49f412c11bb50ca344eb55deb78b0f28454fe8508d22e
Nov 03 20:07:24.320927 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:24.320852    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-594f764c95-z8fjv" event={"ID":"b5520324-5b0c-4417-8221-a2cf1c08b881","Type":"ContainerStarted","Data":"2c22210631cd937734f49f412c11bb50ca344eb55deb78b0f28454fe8508d22e"}
Nov 03 20:07:24.953654 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:24.953619    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jb45k"]
Nov 03 20:07:25.013773 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:25.013742    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jb45k"]
Nov 03 20:07:25.013959 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:25.013841    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jb45k"
Nov 03 20:07:25.023865 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:25.023814    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nh6tb\""
Nov 03 20:07:25.023865 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:25.023817    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 03 20:07:25.023865 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:25.023861    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 20:07:25.024106 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:25.023875    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 20:07:25.024106 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:25.023879    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 03 20:07:25.119444 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:25.119408    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clgt6\" (UniqueName: \"kubernetes.io/projected/7e5c45c5-1f48-4b84-b7ae-244cd6e80b9f-kube-api-access-clgt6\") pod \"kuadrant-console-plugin-66c4fb575d-jb45k\" (UID: \"7e5c45c5-1f48-4b84-b7ae-244cd6e80b9f\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jb45k"
Nov 03 20:07:25.119712 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:25.119459    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7e5c45c5-1f48-4b84-b7ae-244cd6e80b9f-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jb45k\" (UID: \"7e5c45c5-1f48-4b84-b7ae-244cd6e80b9f\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jb45k"
Nov 03 20:07:25.119712 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:25.119549    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e5c45c5-1f48-4b84-b7ae-244cd6e80b9f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jb45k\" (UID: \"7e5c45c5-1f48-4b84-b7ae-244cd6e80b9f\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jb45k"
Nov 03 20:07:25.220402 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:25.220322    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7e5c45c5-1f48-4b84-b7ae-244cd6e80b9f-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jb45k\" (UID: \"7e5c45c5-1f48-4b84-b7ae-244cd6e80b9f\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jb45k"
Nov 03 20:07:25.220402 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:25.220372    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e5c45c5-1f48-4b84-b7ae-244cd6e80b9f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jb45k\" (UID: \"7e5c45c5-1f48-4b84-b7ae-244cd6e80b9f\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jb45k"
Nov 03 20:07:25.220637 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:25.220443    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clgt6\" (UniqueName: \"kubernetes.io/projected/7e5c45c5-1f48-4b84-b7ae-244cd6e80b9f-kube-api-access-clgt6\") pod \"kuadrant-console-plugin-66c4fb575d-jb45k\" (UID: \"7e5c45c5-1f48-4b84-b7ae-244cd6e80b9f\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jb45k"
Nov 03 20:07:25.221131 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:25.221103    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7e5c45c5-1f48-4b84-b7ae-244cd6e80b9f-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jb45k\" (UID: \"7e5c45c5-1f48-4b84-b7ae-244cd6e80b9f\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jb45k"
Nov 03 20:07:25.223082 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:25.223062    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7e5c45c5-1f48-4b84-b7ae-244cd6e80b9f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jb45k\" (UID: \"7e5c45c5-1f48-4b84-b7ae-244cd6e80b9f\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jb45k"
Nov 03 20:07:25.232789 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:25.232759    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clgt6\" (UniqueName: \"kubernetes.io/projected/7e5c45c5-1f48-4b84-b7ae-244cd6e80b9f-kube-api-access-clgt6\") pod \"kuadrant-console-plugin-66c4fb575d-jb45k\" (UID: \"7e5c45c5-1f48-4b84-b7ae-244cd6e80b9f\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jb45k"
Nov 03 20:07:25.351689 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:25.351657    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jb45k"
Nov 03 20:07:25.431410 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:25.431383    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-n49pr"]
Nov 03 20:07:25.435796 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:25.435778    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-n49pr"
Nov 03 20:07:25.442194 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:25.442175    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-n49pr"]
Nov 03 20:07:25.522922 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:25.522855    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfffw\" (UniqueName: \"kubernetes.io/projected/e739b617-6018-48c9-9203-8f19a1213993-kube-api-access-dfffw\") pod \"authorino-56c6c9ddd5-n49pr\" (UID: \"e739b617-6018-48c9-9203-8f19a1213993\") " pod="kuadrant-system/authorino-56c6c9ddd5-n49pr"
Nov 03 20:07:25.523464 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:25.523442    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-44qgz\""
Nov 03 20:07:25.624097 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:25.624059    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfffw\" (UniqueName: \"kubernetes.io/projected/e739b617-6018-48c9-9203-8f19a1213993-kube-api-access-dfffw\") pod \"authorino-56c6c9ddd5-n49pr\" (UID: \"e739b617-6018-48c9-9203-8f19a1213993\") " pod="kuadrant-system/authorino-56c6c9ddd5-n49pr"
Nov 03 20:07:25.628950 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:25.628922    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jb45k"]
Nov 03 20:07:25.632948 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:07:25.632924    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e5c45c5_1f48_4b84_b7ae_244cd6e80b9f.slice/crio-40c85b75d8b4484a868756b6fa87943ad56e285bb83dbb6b4e644921d5ea3e40 WatchSource:0}: Error finding container 40c85b75d8b4484a868756b6fa87943ad56e285bb83dbb6b4e644921d5ea3e40: Status 404 returned error can't find the container with id 40c85b75d8b4484a868756b6fa87943ad56e285bb83dbb6b4e644921d5ea3e40
Nov 03 20:07:25.642398 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:25.642372    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfffw\" (UniqueName: \"kubernetes.io/projected/e739b617-6018-48c9-9203-8f19a1213993-kube-api-access-dfffw\") pod \"authorino-56c6c9ddd5-n49pr\" (UID: \"e739b617-6018-48c9-9203-8f19a1213993\") " pod="kuadrant-system/authorino-56c6c9ddd5-n49pr"
Nov 03 20:07:25.752412 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:25.752371    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-n49pr"
Nov 03 20:07:26.031816 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:26.031786    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-n49pr"]
Nov 03 20:07:26.034671 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:07:26.034646    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode739b617_6018_48c9_9203_8f19a1213993.slice/crio-91572eb684898f1cf4671b22efd0afa6fdf204f5f6a30b61d2741768d56ac0a4 WatchSource:0}: Error finding container 91572eb684898f1cf4671b22efd0afa6fdf204f5f6a30b61d2741768d56ac0a4: Status 404 returned error can't find the container with id 91572eb684898f1cf4671b22efd0afa6fdf204f5f6a30b61d2741768d56ac0a4
Nov 03 20:07:26.329840 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:26.329803    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jb45k" event={"ID":"7e5c45c5-1f48-4b84-b7ae-244cd6e80b9f","Type":"ContainerStarted","Data":"40c85b75d8b4484a868756b6fa87943ad56e285bb83dbb6b4e644921d5ea3e40"}
Nov 03 20:07:26.330927 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:26.330890    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-n49pr" event={"ID":"e739b617-6018-48c9-9203-8f19a1213993","Type":"ContainerStarted","Data":"91572eb684898f1cf4671b22efd0afa6fdf204f5f6a30b61d2741768d56ac0a4"}
Nov 03 20:07:26.332222 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:26.332199    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-594f764c95-z8fjv" event={"ID":"b5520324-5b0c-4417-8221-a2cf1c08b881","Type":"ContainerStarted","Data":"fcba7b35fbdda829364d6a3e59a4a09190dc61baed6ed09b953926f758d6578d"}
Nov 03 20:07:26.350981 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:26.350936    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-594f764c95-z8fjv" podStartSLOduration=1.855348435 podStartE2EDuration="4.350924693s" totalImagesPullingTime="2.495576258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:26.350323593 +0000 UTC m=+376.483452056" watchObservedRunningTime="2026-11-03 20:07:26.350924693 +0000 UTC m=+376.484053154"
Nov 03 20:07:30.351302 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:30.351264    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-n49pr" event={"ID":"e739b617-6018-48c9-9203-8f19a1213993","Type":"ContainerStarted","Data":"fdcaf9e8dd89863b25e421f8eb756bc082b1f155f2c0f3a521b64a7583b2af4e"}
Nov 03 20:07:30.368766 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:30.368707    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-n49pr" podStartSLOduration=2.000782034 podStartE2EDuration="5.368676344s" totalImagesPullingTime="3.36789431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:30.367743836 +0000 UTC m=+380.500872307" watchObservedRunningTime="2026-11-03 20:07:30.368676344 +0000 UTC m=+380.501804814"
Nov 03 20:07:32.361506 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:32.361465    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jb45k" event={"ID":"7e5c45c5-1f48-4b84-b7ae-244cd6e80b9f","Type":"ContainerStarted","Data":"29a0dc1b8346ecd4a5b8a7301528de28f847358985c03f5d7fb63300d4f27bf1"}
Nov 03 20:07:32.379785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:07:32.379736    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jb45k" podStartSLOduration=2.376909896 podStartE2EDuration="8.379721346s" totalImagesPullingTime="6.00281145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:32.378895721 +0000 UTC m=+382.512024186" watchObservedRunningTime="2026-11-03 20:07:32.379721346 +0000 UTC m=+382.512849808"
Nov 03 20:07:49.626742 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:07:49.626667    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:08:05.682703 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:05.682665    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5d77cc995-4992w"]
Nov 03 20:08:05.703972 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:05.703940    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d77cc995-4992w"]
Nov 03 20:08:05.704114 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:05.704057    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d77cc995-4992w"
Nov 03 20:08:05.708003 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:05.707975    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 20:08:05.709193 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:05.709171    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 20:08:05.709193 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:05.709181    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 20:08:05.709391 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:05.709210    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7tsr7\""
Nov 03 20:08:05.709391 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:05.709240    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 20:08:05.709391 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:05.709332    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 20:08:05.739391 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:05.739360    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4b449554-2b61-4b92-91bc-dd4531ba3d8e-webhook-cert\") pod \"maas-controller-5d77cc995-4992w\" (UID: \"4b449554-2b61-4b92-91bc-dd4531ba3d8e\") " pod="opendatahub/maas-controller-5d77cc995-4992w"
Nov 03 20:08:05.739505 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:05.739398    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b449554-2b61-4b92-91bc-dd4531ba3d8e-metrics-certs\") pod \"maas-controller-5d77cc995-4992w\" (UID: \"4b449554-2b61-4b92-91bc-dd4531ba3d8e\") " pod="opendatahub/maas-controller-5d77cc995-4992w"
Nov 03 20:08:05.739505 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:05.739442    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jngjl\" (UniqueName: \"kubernetes.io/projected/4b449554-2b61-4b92-91bc-dd4531ba3d8e-kube-api-access-jngjl\") pod \"maas-controller-5d77cc995-4992w\" (UID: \"4b449554-2b61-4b92-91bc-dd4531ba3d8e\") " pod="opendatahub/maas-controller-5d77cc995-4992w"
Nov 03 20:08:05.840287 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:05.840237    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b449554-2b61-4b92-91bc-dd4531ba3d8e-metrics-certs\") pod \"maas-controller-5d77cc995-4992w\" (UID: \"4b449554-2b61-4b92-91bc-dd4531ba3d8e\") " pod="opendatahub/maas-controller-5d77cc995-4992w"
Nov 03 20:08:05.840474 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:05.840323    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jngjl\" (UniqueName: \"kubernetes.io/projected/4b449554-2b61-4b92-91bc-dd4531ba3d8e-kube-api-access-jngjl\") pod \"maas-controller-5d77cc995-4992w\" (UID: \"4b449554-2b61-4b92-91bc-dd4531ba3d8e\") " pod="opendatahub/maas-controller-5d77cc995-4992w"
Nov 03 20:08:05.840474 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:05.840410    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4b449554-2b61-4b92-91bc-dd4531ba3d8e-webhook-cert\") pod \"maas-controller-5d77cc995-4992w\" (UID: \"4b449554-2b61-4b92-91bc-dd4531ba3d8e\") " pod="opendatahub/maas-controller-5d77cc995-4992w"
Nov 03 20:08:05.842992 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:05.842966    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4b449554-2b61-4b92-91bc-dd4531ba3d8e-webhook-cert\") pod \"maas-controller-5d77cc995-4992w\" (UID: \"4b449554-2b61-4b92-91bc-dd4531ba3d8e\") " pod="opendatahub/maas-controller-5d77cc995-4992w"
Nov 03 20:08:05.843102 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:05.842966    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b449554-2b61-4b92-91bc-dd4531ba3d8e-metrics-certs\") pod \"maas-controller-5d77cc995-4992w\" (UID: \"4b449554-2b61-4b92-91bc-dd4531ba3d8e\") " pod="opendatahub/maas-controller-5d77cc995-4992w"
Nov 03 20:08:05.853068 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:05.853041    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jngjl\" (UniqueName: \"kubernetes.io/projected/4b449554-2b61-4b92-91bc-dd4531ba3d8e-kube-api-access-jngjl\") pod \"maas-controller-5d77cc995-4992w\" (UID: \"4b449554-2b61-4b92-91bc-dd4531ba3d8e\") " pod="opendatahub/maas-controller-5d77cc995-4992w"
Nov 03 20:08:06.017066 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:06.017041    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d77cc995-4992w"
Nov 03 20:08:06.140718 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:06.140683    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-687b69bd9-6n6dr"]
Nov 03 20:08:06.143040 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:06.143022    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-687b69bd9-6n6dr"
Nov 03 20:08:06.154450 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:06.154425    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-687b69bd9-6n6dr"]
Nov 03 20:08:06.243460 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:06.243421    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd8fabe-6f14-458d-ae20-bc4b71333adf-metrics-certs\") pod \"maas-controller-687b69bd9-6n6dr\" (UID: \"2bd8fabe-6f14-458d-ae20-bc4b71333adf\") " pod="opendatahub/maas-controller-687b69bd9-6n6dr"
Nov 03 20:08:06.243460 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:06.243455    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2bd8fabe-6f14-458d-ae20-bc4b71333adf-webhook-cert\") pod \"maas-controller-687b69bd9-6n6dr\" (UID: \"2bd8fabe-6f14-458d-ae20-bc4b71333adf\") " pod="opendatahub/maas-controller-687b69bd9-6n6dr"
Nov 03 20:08:06.243725 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:06.243501    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9pt2\" (UniqueName: \"kubernetes.io/projected/2bd8fabe-6f14-458d-ae20-bc4b71333adf-kube-api-access-v9pt2\") pod \"maas-controller-687b69bd9-6n6dr\" (UID: \"2bd8fabe-6f14-458d-ae20-bc4b71333adf\") " pod="opendatahub/maas-controller-687b69bd9-6n6dr"
Nov 03 20:08:06.290030 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:06.290006    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d77cc995-4992w"]
Nov 03 20:08:06.292536 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:08:06.292505    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b449554_2b61_4b92_91bc_dd4531ba3d8e.slice/crio-77b11dd2eac6b1dd26ae0b347559928dcdd7ee35f183a903ec1b8e22f1e7bc72 WatchSource:0}: Error finding container 77b11dd2eac6b1dd26ae0b347559928dcdd7ee35f183a903ec1b8e22f1e7bc72: Status 404 returned error can't find the container with id 77b11dd2eac6b1dd26ae0b347559928dcdd7ee35f183a903ec1b8e22f1e7bc72
Nov 03 20:08:06.344864 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:06.344842    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9pt2\" (UniqueName: \"kubernetes.io/projected/2bd8fabe-6f14-458d-ae20-bc4b71333adf-kube-api-access-v9pt2\") pod \"maas-controller-687b69bd9-6n6dr\" (UID: \"2bd8fabe-6f14-458d-ae20-bc4b71333adf\") " pod="opendatahub/maas-controller-687b69bd9-6n6dr"
Nov 03 20:08:06.344978 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:06.344887    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd8fabe-6f14-458d-ae20-bc4b71333adf-metrics-certs\") pod \"maas-controller-687b69bd9-6n6dr\" (UID: \"2bd8fabe-6f14-458d-ae20-bc4b71333adf\") " pod="opendatahub/maas-controller-687b69bd9-6n6dr"
Nov 03 20:08:06.344978 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:06.344909    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2bd8fabe-6f14-458d-ae20-bc4b71333adf-webhook-cert\") pod \"maas-controller-687b69bd9-6n6dr\" (UID: \"2bd8fabe-6f14-458d-ae20-bc4b71333adf\") " pod="opendatahub/maas-controller-687b69bd9-6n6dr"
Nov 03 20:08:06.347420 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:06.347401    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2bd8fabe-6f14-458d-ae20-bc4b71333adf-webhook-cert\") pod \"maas-controller-687b69bd9-6n6dr\" (UID: \"2bd8fabe-6f14-458d-ae20-bc4b71333adf\") " pod="opendatahub/maas-controller-687b69bd9-6n6dr"
Nov 03 20:08:06.347520 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:06.347446    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd8fabe-6f14-458d-ae20-bc4b71333adf-metrics-certs\") pod \"maas-controller-687b69bd9-6n6dr\" (UID: \"2bd8fabe-6f14-458d-ae20-bc4b71333adf\") " pod="opendatahub/maas-controller-687b69bd9-6n6dr"
Nov 03 20:08:06.357392 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:06.357374    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9pt2\" (UniqueName: \"kubernetes.io/projected/2bd8fabe-6f14-458d-ae20-bc4b71333adf-kube-api-access-v9pt2\") pod \"maas-controller-687b69bd9-6n6dr\" (UID: \"2bd8fabe-6f14-458d-ae20-bc4b71333adf\") " pod="opendatahub/maas-controller-687b69bd9-6n6dr"
Nov 03 20:08:06.454731 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:06.454688    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-687b69bd9-6n6dr"
Nov 03 20:08:06.486116 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:06.486086    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d77cc995-4992w" event={"ID":"4b449554-2b61-4b92-91bc-dd4531ba3d8e","Type":"ContainerStarted","Data":"77b11dd2eac6b1dd26ae0b347559928dcdd7ee35f183a903ec1b8e22f1e7bc72"}
Nov 03 20:08:06.732306 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:06.729968    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-687b69bd9-6n6dr"]
Nov 03 20:08:06.733211 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:08:06.733179    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd8fabe_6f14_458d_ae20_bc4b71333adf.slice/crio-ad5c891e7279b2c3cdab83108c40e12f1fd3a1cc68203c67452325d41852f7c9 WatchSource:0}: Error finding container ad5c891e7279b2c3cdab83108c40e12f1fd3a1cc68203c67452325d41852f7c9: Status 404 returned error can't find the container with id ad5c891e7279b2c3cdab83108c40e12f1fd3a1cc68203c67452325d41852f7c9
Nov 03 20:08:07.491857 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:07.491822    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-687b69bd9-6n6dr" event={"ID":"2bd8fabe-6f14-458d-ae20-bc4b71333adf","Type":"ContainerStarted","Data":"ad5c891e7279b2c3cdab83108c40e12f1fd3a1cc68203c67452325d41852f7c9"}
Nov 03 20:08:10.804879 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:10.804858    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 20:08:11.512252 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:11.512211    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d77cc995-4992w" event={"ID":"4b449554-2b61-4b92-91bc-dd4531ba3d8e","Type":"ContainerStarted","Data":"70770412cdcc264a5b2066534ad21ca6417fb2c217de25164e9a7f7b28a3d8de"}
Nov 03 20:08:11.512470 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:11.512319    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5d77cc995-4992w"
Nov 03 20:08:11.513729 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:11.513707    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-687b69bd9-6n6dr" event={"ID":"2bd8fabe-6f14-458d-ae20-bc4b71333adf","Type":"ContainerStarted","Data":"320f29b71fa69d9f2d721bcc3281c3f9c573e68e059a49d5f5cb9f49bca74437"}
Nov 03 20:08:11.513854 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:11.513838    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-687b69bd9-6n6dr"
Nov 03 20:08:11.533459 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:11.533381    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5d77cc995-4992w" podStartSLOduration=2.025745406 podStartE2EDuration="6.533366336s" totalImagesPullingTime="4.50762093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:11.533141788 +0000 UTC m=+421.666270251" watchObservedRunningTime="2026-11-03 20:08:11.533366336 +0000 UTC m=+421.666494799"
Nov 03 20:08:11.553864 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:11.553794    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-687b69bd9-6n6dr" podStartSLOduration=1.479183286 podStartE2EDuration="5.553778334s" totalImagesPullingTime="4.074595048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:11.552717314 +0000 UTC m=+421.685845779" watchObservedRunningTime="2026-11-03 20:08:11.553778334 +0000 UTC m=+421.686906797"
Nov 03 20:08:12.113383 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:12.113350    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4bbvd"]
Nov 03 20:08:12.117006 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:12.116972    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4bbvd"
Nov 03 20:08:12.126106 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:12.126081    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 03 20:08:12.126264 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:12.126081    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 03 20:08:12.126264 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:12.126141    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-v279t\""
Nov 03 20:08:12.127413 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:12.127396    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 03 20:08:12.138001 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:12.137976    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4bbvd"]
Nov 03 20:08:12.196696 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:12.196661    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1dcbd7f2-5145-444c-9b8f-7035eb14b64e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4bbvd\" (UID: \"1dcbd7f2-5145-444c-9b8f-7035eb14b64e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4bbvd"
Nov 03 20:08:12.196696 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:12.196702    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp4px\" (UniqueName: \"kubernetes.io/projected/1dcbd7f2-5145-444c-9b8f-7035eb14b64e-kube-api-access-cp4px\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4bbvd\" (UID: \"1dcbd7f2-5145-444c-9b8f-7035eb14b64e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4bbvd"
Nov 03 20:08:12.196922 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:12.196785    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1dcbd7f2-5145-444c-9b8f-7035eb14b64e-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4bbvd\" (UID: \"1dcbd7f2-5145-444c-9b8f-7035eb14b64e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4bbvd"
Nov 03 20:08:12.196922 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:12.196880    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/1dcbd7f2-5145-444c-9b8f-7035eb14b64e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4bbvd\" (UID: \"1dcbd7f2-5145-444c-9b8f-7035eb14b64e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4bbvd"
Nov 03 20:08:12.297814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:12.297778    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1dcbd7f2-5145-444c-9b8f-7035eb14b64e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4bbvd\" (UID: \"1dcbd7f2-5145-444c-9b8f-7035eb14b64e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4bbvd"
Nov 03 20:08:12.297814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:12.297816    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp4px\" (UniqueName: \"kubernetes.io/projected/1dcbd7f2-5145-444c-9b8f-7035eb14b64e-kube-api-access-cp4px\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4bbvd\" (UID: \"1dcbd7f2-5145-444c-9b8f-7035eb14b64e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4bbvd"
Nov 03 20:08:12.298026 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:12.297845    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1dcbd7f2-5145-444c-9b8f-7035eb14b64e-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4bbvd\" (UID: \"1dcbd7f2-5145-444c-9b8f-7035eb14b64e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4bbvd"
Nov 03 20:08:12.298026 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:12.297869    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/1dcbd7f2-5145-444c-9b8f-7035eb14b64e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4bbvd\" (UID: \"1dcbd7f2-5145-444c-9b8f-7035eb14b64e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4bbvd"
Nov 03 20:08:12.298541 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:12.298523    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1dcbd7f2-5145-444c-9b8f-7035eb14b64e-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4bbvd\" (UID: \"1dcbd7f2-5145-444c-9b8f-7035eb14b64e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4bbvd"
Nov 03 20:08:12.298695 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:12.298679    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/1dcbd7f2-5145-444c-9b8f-7035eb14b64e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4bbvd\" (UID: \"1dcbd7f2-5145-444c-9b8f-7035eb14b64e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4bbvd"
Nov 03 20:08:12.300643 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:12.300624    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1dcbd7f2-5145-444c-9b8f-7035eb14b64e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4bbvd\" (UID: \"1dcbd7f2-5145-444c-9b8f-7035eb14b64e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4bbvd"
Nov 03 20:08:12.310906 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:12.310883    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp4px\" (UniqueName: \"kubernetes.io/projected/1dcbd7f2-5145-444c-9b8f-7035eb14b64e-kube-api-access-cp4px\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4bbvd\" (UID: \"1dcbd7f2-5145-444c-9b8f-7035eb14b64e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4bbvd"
Nov 03 20:08:12.432462 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:12.432389    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4bbvd"
Nov 03 20:08:12.729472 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:12.729384    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4bbvd"]
Nov 03 20:08:12.733167 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:08:12.733136    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dcbd7f2_5145_444c_9b8f_7035eb14b64e.slice/crio-6d7338fd9348fc7deddc7bccff2a32e2809d0a4b8fb3ef52ece0b441112e8007 WatchSource:0}: Error finding container 6d7338fd9348fc7deddc7bccff2a32e2809d0a4b8fb3ef52ece0b441112e8007: Status 404 returned error can't find the container with id 6d7338fd9348fc7deddc7bccff2a32e2809d0a4b8fb3ef52ece0b441112e8007
Nov 03 20:08:13.523895 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:13.523862    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4bbvd" event={"ID":"1dcbd7f2-5145-444c-9b8f-7035eb14b64e","Type":"ContainerStarted","Data":"6d7338fd9348fc7deddc7bccff2a32e2809d0a4b8fb3ef52ece0b441112e8007"}
Nov 03 20:08:18.853163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:18.853127    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-2725f"]
Nov 03 20:08:18.856679 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:18.856657    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-2725f"
Nov 03 20:08:18.864191 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:18.864171    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 03 20:08:18.865048 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:18.865018    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-tg2kd\""
Nov 03 20:08:18.877281 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:18.877204    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-2725f"]
Nov 03 20:08:18.906621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:18.906568    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-8lzxb"]
Nov 03 20:08:18.911126 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:18.910388    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-8lzxb"
Nov 03 20:08:18.920019 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:18.919990    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-99ntv"]
Nov 03 20:08:18.923457 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:18.923433    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-99ntv"
Nov 03 20:08:18.926670 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:18.926646    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-8lzxb"]
Nov 03 20:08:18.951651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:18.950536    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 03 20:08:18.951651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:18.950699    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 03 20:08:18.951651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:18.950833    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-7kr7t\""
Nov 03 20:08:18.951651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:18.950952    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2794w\""
Nov 03 20:08:18.951651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:18.951060    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 03 20:08:18.951959 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:18.951922    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-99ntv"]
Nov 03 20:08:18.979847 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:18.979817    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c4bb3bf-caf2-4932-8b65-00189e8d41fb-cert\") pod \"kserve-controller-manager-776cc66d47-2725f\" (UID: \"1c4bb3bf-caf2-4932-8b65-00189e8d41fb\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2725f"
Nov 03 20:08:18.980416 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:18.979871    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ljzd\" (UniqueName: \"kubernetes.io/projected/1c4bb3bf-caf2-4932-8b65-00189e8d41fb-kube-api-access-9ljzd\") pod \"kserve-controller-manager-776cc66d47-2725f\" (UID: \"1c4bb3bf-caf2-4932-8b65-00189e8d41fb\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2725f"
Nov 03 20:08:19.081453 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.081173    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c4bb3bf-caf2-4932-8b65-00189e8d41fb-cert\") pod \"kserve-controller-manager-776cc66d47-2725f\" (UID: \"1c4bb3bf-caf2-4932-8b65-00189e8d41fb\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2725f"
Nov 03 20:08:19.081453 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.081232    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ljzd\" (UniqueName: \"kubernetes.io/projected/1c4bb3bf-caf2-4932-8b65-00189e8d41fb-kube-api-access-9ljzd\") pod \"kserve-controller-manager-776cc66d47-2725f\" (UID: \"1c4bb3bf-caf2-4932-8b65-00189e8d41fb\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2725f"
Nov 03 20:08:19.081453 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.081274    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs8pd\" (UniqueName: \"kubernetes.io/projected/c23a53b9-ca79-461a-a8f6-20618ffc35aa-kube-api-access-cs8pd\") pod \"model-serving-api-75bd89fdfc-8lzxb\" (UID: \"c23a53b9-ca79-461a-a8f6-20618ffc35aa\") " pod="opendatahub/model-serving-api-75bd89fdfc-8lzxb"
Nov 03 20:08:19.081453 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.081334    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c23a53b9-ca79-461a-a8f6-20618ffc35aa-tls-certs\") pod \"model-serving-api-75bd89fdfc-8lzxb\" (UID: \"c23a53b9-ca79-461a-a8f6-20618ffc35aa\") " pod="opendatahub/model-serving-api-75bd89fdfc-8lzxb"
Nov 03 20:08:19.081453 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.081361    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b347c3ff-a128-4fd2-8c07-dfea5c60a431-cert\") pod \"odh-model-controller-769875db56-99ntv\" (UID: \"b347c3ff-a128-4fd2-8c07-dfea5c60a431\") " pod="opendatahub/odh-model-controller-769875db56-99ntv"
Nov 03 20:08:19.081453 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.081383    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b347c3ff-a128-4fd2-8c07-dfea5c60a431-metrics-certs\") pod \"odh-model-controller-769875db56-99ntv\" (UID: \"b347c3ff-a128-4fd2-8c07-dfea5c60a431\") " pod="opendatahub/odh-model-controller-769875db56-99ntv"
Nov 03 20:08:19.081874 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.081477    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxrbv\" (UniqueName: \"kubernetes.io/projected/b347c3ff-a128-4fd2-8c07-dfea5c60a431-kube-api-access-fxrbv\") pod \"odh-model-controller-769875db56-99ntv\" (UID: \"b347c3ff-a128-4fd2-8c07-dfea5c60a431\") " pod="opendatahub/odh-model-controller-769875db56-99ntv"
Nov 03 20:08:19.084880 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.084851    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c4bb3bf-caf2-4932-8b65-00189e8d41fb-cert\") pod \"kserve-controller-manager-776cc66d47-2725f\" (UID: \"1c4bb3bf-caf2-4932-8b65-00189e8d41fb\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2725f"
Nov 03 20:08:19.094084 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.094055    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ljzd\" (UniqueName: \"kubernetes.io/projected/1c4bb3bf-caf2-4932-8b65-00189e8d41fb-kube-api-access-9ljzd\") pod \"kserve-controller-manager-776cc66d47-2725f\" (UID: \"1c4bb3bf-caf2-4932-8b65-00189e8d41fb\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2725f"
Nov 03 20:08:19.179582 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.179499    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-2725f"
Nov 03 20:08:19.183040 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.182648    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs8pd\" (UniqueName: \"kubernetes.io/projected/c23a53b9-ca79-461a-a8f6-20618ffc35aa-kube-api-access-cs8pd\") pod \"model-serving-api-75bd89fdfc-8lzxb\" (UID: \"c23a53b9-ca79-461a-a8f6-20618ffc35aa\") " pod="opendatahub/model-serving-api-75bd89fdfc-8lzxb"
Nov 03 20:08:19.183040 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.182717    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c23a53b9-ca79-461a-a8f6-20618ffc35aa-tls-certs\") pod \"model-serving-api-75bd89fdfc-8lzxb\" (UID: \"c23a53b9-ca79-461a-a8f6-20618ffc35aa\") " pod="opendatahub/model-serving-api-75bd89fdfc-8lzxb"
Nov 03 20:08:19.183040 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.182741    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b347c3ff-a128-4fd2-8c07-dfea5c60a431-cert\") pod \"odh-model-controller-769875db56-99ntv\" (UID: \"b347c3ff-a128-4fd2-8c07-dfea5c60a431\") " pod="opendatahub/odh-model-controller-769875db56-99ntv"
Nov 03 20:08:19.183040 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.182764    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b347c3ff-a128-4fd2-8c07-dfea5c60a431-metrics-certs\") pod \"odh-model-controller-769875db56-99ntv\" (UID: \"b347c3ff-a128-4fd2-8c07-dfea5c60a431\") " pod="opendatahub/odh-model-controller-769875db56-99ntv"
Nov 03 20:08:19.183040 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.182797    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxrbv\" (UniqueName: \"kubernetes.io/projected/b347c3ff-a128-4fd2-8c07-dfea5c60a431-kube-api-access-fxrbv\") pod \"odh-model-controller-769875db56-99ntv\" (UID: \"b347c3ff-a128-4fd2-8c07-dfea5c60a431\") " pod="opendatahub/odh-model-controller-769875db56-99ntv"
Nov 03 20:08:19.186873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.186824    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c23a53b9-ca79-461a-a8f6-20618ffc35aa-tls-certs\") pod \"model-serving-api-75bd89fdfc-8lzxb\" (UID: \"c23a53b9-ca79-461a-a8f6-20618ffc35aa\") " pod="opendatahub/model-serving-api-75bd89fdfc-8lzxb"
Nov 03 20:08:19.187476 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.187456    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b347c3ff-a128-4fd2-8c07-dfea5c60a431-cert\") pod \"odh-model-controller-769875db56-99ntv\" (UID: \"b347c3ff-a128-4fd2-8c07-dfea5c60a431\") " pod="opendatahub/odh-model-controller-769875db56-99ntv"
Nov 03 20:08:19.188833 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.188812    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b347c3ff-a128-4fd2-8c07-dfea5c60a431-metrics-certs\") pod \"odh-model-controller-769875db56-99ntv\" (UID: \"b347c3ff-a128-4fd2-8c07-dfea5c60a431\") " pod="opendatahub/odh-model-controller-769875db56-99ntv"
Nov 03 20:08:19.195936 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.195908    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxrbv\" (UniqueName: \"kubernetes.io/projected/b347c3ff-a128-4fd2-8c07-dfea5c60a431-kube-api-access-fxrbv\") pod \"odh-model-controller-769875db56-99ntv\" (UID: \"b347c3ff-a128-4fd2-8c07-dfea5c60a431\") " pod="opendatahub/odh-model-controller-769875db56-99ntv"
Nov 03 20:08:19.196528 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.196505    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs8pd\" (UniqueName: \"kubernetes.io/projected/c23a53b9-ca79-461a-a8f6-20618ffc35aa-kube-api-access-cs8pd\") pod \"model-serving-api-75bd89fdfc-8lzxb\" (UID: \"c23a53b9-ca79-461a-a8f6-20618ffc35aa\") " pod="opendatahub/model-serving-api-75bd89fdfc-8lzxb"
Nov 03 20:08:19.237508 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.237473    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"]
Nov 03 20:08:19.237705 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.237585    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-8lzxb"
Nov 03 20:08:19.242012 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.241864    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"
Nov 03 20:08:19.266883 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.257575    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"]
Nov 03 20:08:19.266883 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.262227    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-99ntv"
Nov 03 20:08:19.385354 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.385321    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9pwj7\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"
Nov 03 20:08:19.385518 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.385367    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9pwj7\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"
Nov 03 20:08:19.385561 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.385514    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9pwj7\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"
Nov 03 20:08:19.385561 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.385550    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7b3aa674-8eed-440f-af1e-94112054a6f1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9pwj7\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"
Nov 03 20:08:19.385696 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.385579    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlzcg\" (UniqueName: \"kubernetes.io/projected/7b3aa674-8eed-440f-af1e-94112054a6f1-kube-api-access-dlzcg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9pwj7\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"
Nov 03 20:08:19.385770 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.385716    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9pwj7\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"
Nov 03 20:08:19.385880 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.385809    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b3aa674-8eed-440f-af1e-94112054a6f1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9pwj7\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"
Nov 03 20:08:19.484132 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.483984    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-2725f"]
Nov 03 20:08:19.486487 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.486457    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9pwj7\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"
Nov 03 20:08:19.486608 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.486507    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7b3aa674-8eed-440f-af1e-94112054a6f1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9pwj7\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"
Nov 03 20:08:19.486608 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.486556    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlzcg\" (UniqueName: \"kubernetes.io/projected/7b3aa674-8eed-440f-af1e-94112054a6f1-kube-api-access-dlzcg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9pwj7\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"
Nov 03 20:08:19.486721 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.486583    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9pwj7\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"
Nov 03 20:08:19.486721 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.486670    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b3aa674-8eed-440f-af1e-94112054a6f1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9pwj7\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"
Nov 03 20:08:19.486721 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.486706    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9pwj7\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"
Nov 03 20:08:19.486874 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.486738    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9pwj7\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"
Nov 03 20:08:19.487447 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.487422    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9pwj7\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"
Nov 03 20:08:19.494785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.490971    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7b3aa674-8eed-440f-af1e-94112054a6f1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9pwj7\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"
Nov 03 20:08:19.494785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.491326    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9pwj7\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"
Nov 03 20:08:19.494785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.492472    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b3aa674-8eed-440f-af1e-94112054a6f1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9pwj7\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"
Nov 03 20:08:19.500350 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.500317    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9pwj7\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"
Nov 03 20:08:19.501587 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.501565    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9pwj7\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"
Nov 03 20:08:19.505974 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.505949    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlzcg\" (UniqueName: \"kubernetes.io/projected/7b3aa674-8eed-440f-af1e-94112054a6f1-kube-api-access-dlzcg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9pwj7\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"
Nov 03 20:08:19.540544 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.540520    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-8lzxb"]
Nov 03 20:08:19.543445 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:08:19.543422    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc23a53b9_ca79_461a_a8f6_20618ffc35aa.slice/crio-4eccbc6cb12a5daa8a71756d88a034d6c7af88a56ac8a70fdfc2a89acdd726a3 WatchSource:0}: Error finding container 4eccbc6cb12a5daa8a71756d88a034d6c7af88a56ac8a70fdfc2a89acdd726a3: Status 404 returned error can't find the container with id 4eccbc6cb12a5daa8a71756d88a034d6c7af88a56ac8a70fdfc2a89acdd726a3
Nov 03 20:08:19.549920 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.549894    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-2725f" event={"ID":"1c4bb3bf-caf2-4932-8b65-00189e8d41fb","Type":"ContainerStarted","Data":"9fe6c98ea710861fb59e48fa6a77f42caa566e494f603e5be7e30eb373194be4"}
Nov 03 20:08:19.551203 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.551177    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-8lzxb" event={"ID":"c23a53b9-ca79-461a-a8f6-20618ffc35aa","Type":"ContainerStarted","Data":"4eccbc6cb12a5daa8a71756d88a034d6c7af88a56ac8a70fdfc2a89acdd726a3"}
Nov 03 20:08:19.572080 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.572060    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-99ntv"]
Nov 03 20:08:19.574201 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:08:19.574176    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb347c3ff_a128_4fd2_8c07_dfea5c60a431.slice/crio-be95a68374dad7453f5c9ccb466bf2e0172c8586f296c80109f83f25512958e1 WatchSource:0}: Error finding container be95a68374dad7453f5c9ccb466bf2e0172c8586f296c80109f83f25512958e1: Status 404 returned error can't find the container with id be95a68374dad7453f5c9ccb466bf2e0172c8586f296c80109f83f25512958e1
Nov 03 20:08:19.578267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.578250    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"
Nov 03 20:08:19.637885 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:08:19.637845    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:08:19.872525 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.872495    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"]
Nov 03 20:08:19.874787 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:08:19.874754    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b3aa674_8eed_440f_af1e_94112054a6f1.slice/crio-c8cde489768aa74d93bbafb9d79c79fdc910f84149388aff71ec3ece9b3fc2a8 WatchSource:0}: Error finding container c8cde489768aa74d93bbafb9d79c79fdc910f84149388aff71ec3ece9b3fc2a8: Status 404 returned error can't find the container with id c8cde489768aa74d93bbafb9d79c79fdc910f84149388aff71ec3ece9b3fc2a8
Nov 03 20:08:19.877463 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.877417    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:08:19.877570 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:19.877484    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:08:20.565859 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:20.564720    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7" event={"ID":"7b3aa674-8eed-440f-af1e-94112054a6f1","Type":"ContainerStarted","Data":"37c5f9a08a41e0d5185a9ff474e35d35f61f87c28b131008a51363e2e3dd64c2"}
Nov 03 20:08:20.565859 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:20.564780    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7" event={"ID":"7b3aa674-8eed-440f-af1e-94112054a6f1","Type":"ContainerStarted","Data":"c8cde489768aa74d93bbafb9d79c79fdc910f84149388aff71ec3ece9b3fc2a8"}
Nov 03 20:08:20.565859 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:20.565794    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"
Nov 03 20:08:20.567590 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:20.567434    4421 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-9pwj7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:08:20.567590 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:20.567510    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7" podUID="7b3aa674-8eed-440f-af1e-94112054a6f1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:08:20.568975 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:20.568921    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-99ntv" event={"ID":"b347c3ff-a128-4fd2-8c07-dfea5c60a431","Type":"ContainerStarted","Data":"be95a68374dad7453f5c9ccb466bf2e0172c8586f296c80109f83f25512958e1"}
Nov 03 20:08:20.620135 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:20.619897    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7" podStartSLOduration=1.619878449 podStartE2EDuration="1.619878449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:20.616402905 +0000 UTC m=+430.749531370" watchObservedRunningTime="2026-11-03 20:08:20.619878449 +0000 UTC m=+430.753006915"
Nov 03 20:08:21.578007 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:21.577973    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"
Nov 03 20:08:21.634291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:21.634255    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"]
Nov 03 20:08:21.634532 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:21.634494    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs" podUID="12e64430-b367-4411-b366-d7c0f6e19eaf" containerName="discovery" containerID="cri-o://b33046188ecb1076b70ef0188b25ba7a61a81a9b2b1c173bea1372e1a023991c" gracePeriod=30
Nov 03 20:08:21.981702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:21.981674    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"
Nov 03 20:08:22.138173 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.137888    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-csr-dns-cert\") pod \"12e64430-b367-4411-b366-d7c0f6e19eaf\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") "
Nov 03 20:08:22.138173 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.137970    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-kubeconfig\") pod \"12e64430-b367-4411-b366-d7c0f6e19eaf\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") "
Nov 03 20:08:22.138173 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.138131    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12e64430-b367-4411-b366-d7c0f6e19eaf-kube-api-access-qmwc6\" (UniqueName: \"kubernetes.io/projected/12e64430-b367-4411-b366-d7c0f6e19eaf-kube-api-access-qmwc6\") pod \"12e64430-b367-4411-b366-d7c0f6e19eaf\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") "
Nov 03 20:08:22.138439 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.138199    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-csr-ca-configmap\") pod \"12e64430-b367-4411-b366-d7c0f6e19eaf\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") "
Nov 03 20:08:22.138439 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.138228    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12e64430-b367-4411-b366-d7c0f6e19eaf-cacerts\" (UniqueName: \"kubernetes.io/secret/12e64430-b367-4411-b366-d7c0f6e19eaf-cacerts\") pod \"12e64430-b367-4411-b366-d7c0f6e19eaf\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") "
Nov 03 20:08:22.138439 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.138356    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-token\" (UniqueName: \"kubernetes.io/projected/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-token\") pod \"12e64430-b367-4411-b366-d7c0f6e19eaf\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") "
Nov 03 20:08:22.138439 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.138384    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12e64430-b367-4411-b366-d7c0f6e19eaf-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e64430-b367-4411-b366-d7c0f6e19eaf-local-certs\") pod \"12e64430-b367-4411-b366-d7c0f6e19eaf\" (UID: \"12e64430-b367-4411-b366-d7c0f6e19eaf\") "
Nov 03 20:08:22.139800 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.139772    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-csr-ca-configmap" pod "12e64430-b367-4411-b366-d7c0f6e19eaf" (UID: "12e64430-b367-4411-b366-d7c0f6e19eaf"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:22.144227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.144189    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-kubeconfig" pod "12e64430-b367-4411-b366-d7c0f6e19eaf" (UID: "12e64430-b367-4411-b366-d7c0f6e19eaf"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:22.147696 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.147655    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e64430-b367-4411-b366-d7c0f6e19eaf-kube-api-access-qmwc6" pod "12e64430-b367-4411-b366-d7c0f6e19eaf" (UID: "12e64430-b367-4411-b366-d7c0f6e19eaf"). InnerVolumeSpecName "kube-api-access-qmwc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:22.148813 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.148767    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-csr-dns-cert" pod "12e64430-b367-4411-b366-d7c0f6e19eaf" (UID: "12e64430-b367-4411-b366-d7c0f6e19eaf"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:22.150198 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.150118    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e64430-b367-4411-b366-d7c0f6e19eaf-local-certs" pod "12e64430-b367-4411-b366-d7c0f6e19eaf" (UID: "12e64430-b367-4411-b366-d7c0f6e19eaf"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:22.150198 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.150176    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-token" pod "12e64430-b367-4411-b366-d7c0f6e19eaf" (UID: "12e64430-b367-4411-b366-d7c0f6e19eaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:22.151131 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.151098    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12e64430-b367-4411-b366-d7c0f6e19eaf-cacerts" pod "12e64430-b367-4411-b366-d7c0f6e19eaf" (UID: "12e64430-b367-4411-b366-d7c0f6e19eaf"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:22.239710 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.239556    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmwc6\" (UniqueName: \"kubernetes.io/projected/12e64430-b367-4411-b366-d7c0f6e19eaf-kube-api-access-qmwc6\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:22.239710 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.239617    4421 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-csr-ca-configmap\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:22.239710 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.239632    4421 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/12e64430-b367-4411-b366-d7c0f6e19eaf-cacerts\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:22.239710 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.239647    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:22.239710 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.239659    4421 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/12e64430-b367-4411-b366-d7c0f6e19eaf-local-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:22.239710 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.239671    4421 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-csr-dns-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:22.239710 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.239682    4421 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12e64430-b367-4411-b366-d7c0f6e19eaf-istio-kubeconfig\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:22.522471 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.521581    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5d77cc995-4992w"
Nov 03 20:08:22.523464 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.522899    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-687b69bd9-6n6dr"
Nov 03 20:08:22.596110 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.596077    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d77cc995-4992w"]
Nov 03 20:08:22.601477 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.601430    4421 generic.go:356] "Generic (PLEG): container finished" podID="12e64430-b367-4411-b366-d7c0f6e19eaf" containerID="b33046188ecb1076b70ef0188b25ba7a61a81a9b2b1c173bea1372e1a023991c" exitCode=0
Nov 03 20:08:22.603521 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.603113    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"
Nov 03 20:08:22.604385 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.604359    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs" event={"ID":"12e64430-b367-4411-b366-d7c0f6e19eaf","Type":"ContainerDied","Data":"b33046188ecb1076b70ef0188b25ba7a61a81a9b2b1c173bea1372e1a023991c"}
Nov 03 20:08:22.604672 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.604653    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs" event={"ID":"12e64430-b367-4411-b366-d7c0f6e19eaf","Type":"ContainerDied","Data":"22bac99d91f7a7aa390e18b91e7a8aebe7edaebd98801a912738e778d7cd3039"}
Nov 03 20:08:22.604886 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.604871    4421 scope.go:122] "RemoveContainer" containerID="b33046188ecb1076b70ef0188b25ba7a61a81a9b2b1c173bea1372e1a023991c"
Nov 03 20:08:22.608190 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.608167    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5d77cc995-4992w" podUID="4b449554-2b61-4b92-91bc-dd4531ba3d8e" containerName="manager" containerID="cri-o://70770412cdcc264a5b2066534ad21ca6417fb2c217de25164e9a7f7b28a3d8de" gracePeriod=30
Nov 03 20:08:22.641059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.640948    4421 scope.go:122] "RemoveContainer" containerID="b33046188ecb1076b70ef0188b25ba7a61a81a9b2b1c173bea1372e1a023991c"
Nov 03 20:08:22.649561 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.649456    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"]
Nov 03 20:08:22.653162 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.653138    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-kw7bs"]
Nov 03 20:08:22.653560 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:08:22.653539    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b33046188ecb1076b70ef0188b25ba7a61a81a9b2b1c173bea1372e1a023991c\": container with ID starting with b33046188ecb1076b70ef0188b25ba7a61a81a9b2b1c173bea1372e1a023991c not found: ID does not exist" containerID="b33046188ecb1076b70ef0188b25ba7a61a81a9b2b1c173bea1372e1a023991c"
Nov 03 20:08:22.653750 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.653720    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b33046188ecb1076b70ef0188b25ba7a61a81a9b2b1c173bea1372e1a023991c"} err="failed to get container status \"b33046188ecb1076b70ef0188b25ba7a61a81a9b2b1c173bea1372e1a023991c\": rpc error: code = NotFound desc = could not find container \"b33046188ecb1076b70ef0188b25ba7a61a81a9b2b1c173bea1372e1a023991c\": container with ID starting with b33046188ecb1076b70ef0188b25ba7a61a81a9b2b1c173bea1372e1a023991c not found: ID does not exist"
Nov 03 20:08:22.981477 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:22.981196    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d77cc995-4992w"
Nov 03 20:08:23.152838 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:23.151794    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b449554-2b61-4b92-91bc-dd4531ba3d8e-kube-api-access-jngjl\" (UniqueName: \"kubernetes.io/projected/4b449554-2b61-4b92-91bc-dd4531ba3d8e-kube-api-access-jngjl\") pod \"4b449554-2b61-4b92-91bc-dd4531ba3d8e\" (UID: \"4b449554-2b61-4b92-91bc-dd4531ba3d8e\") "
Nov 03 20:08:23.152838 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:23.151926    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b449554-2b61-4b92-91bc-dd4531ba3d8e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b449554-2b61-4b92-91bc-dd4531ba3d8e-metrics-certs\") pod \"4b449554-2b61-4b92-91bc-dd4531ba3d8e\" (UID: \"4b449554-2b61-4b92-91bc-dd4531ba3d8e\") "
Nov 03 20:08:23.152838 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:23.151980    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b449554-2b61-4b92-91bc-dd4531ba3d8e-webhook-cert\" (UniqueName: \"kubernetes.io/secret/4b449554-2b61-4b92-91bc-dd4531ba3d8e-webhook-cert\") pod \"4b449554-2b61-4b92-91bc-dd4531ba3d8e\" (UID: \"4b449554-2b61-4b92-91bc-dd4531ba3d8e\") "
Nov 03 20:08:23.155372 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:23.155325    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b449554-2b61-4b92-91bc-dd4531ba3d8e-webhook-cert" pod "4b449554-2b61-4b92-91bc-dd4531ba3d8e" (UID: "4b449554-2b61-4b92-91bc-dd4531ba3d8e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:23.173572 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:23.173397    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b449554-2b61-4b92-91bc-dd4531ba3d8e-metrics-certs" pod "4b449554-2b61-4b92-91bc-dd4531ba3d8e" (UID: "4b449554-2b61-4b92-91bc-dd4531ba3d8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:23.173572 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:23.173525    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b449554-2b61-4b92-91bc-dd4531ba3d8e-kube-api-access-jngjl" pod "4b449554-2b61-4b92-91bc-dd4531ba3d8e" (UID: "4b449554-2b61-4b92-91bc-dd4531ba3d8e"). InnerVolumeSpecName "kube-api-access-jngjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:23.253245 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:23.253185    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b449554-2b61-4b92-91bc-dd4531ba3d8e-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:23.253245 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:23.253212    4421 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4b449554-2b61-4b92-91bc-dd4531ba3d8e-webhook-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:23.253245 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:23.253227    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jngjl\" (UniqueName: \"kubernetes.io/projected/4b449554-2b61-4b92-91bc-dd4531ba3d8e-kube-api-access-jngjl\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:23.615624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:23.611033    4421 generic.go:356] "Generic (PLEG): container finished" podID="4b449554-2b61-4b92-91bc-dd4531ba3d8e" containerID="70770412cdcc264a5b2066534ad21ca6417fb2c217de25164e9a7f7b28a3d8de" exitCode=0
Nov 03 20:08:23.615624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:23.611222    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d77cc995-4992w" event={"ID":"4b449554-2b61-4b92-91bc-dd4531ba3d8e","Type":"ContainerDied","Data":"70770412cdcc264a5b2066534ad21ca6417fb2c217de25164e9a7f7b28a3d8de"}
Nov 03 20:08:23.615624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:23.611278    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d77cc995-4992w" event={"ID":"4b449554-2b61-4b92-91bc-dd4531ba3d8e","Type":"ContainerDied","Data":"77b11dd2eac6b1dd26ae0b347559928dcdd7ee35f183a903ec1b8e22f1e7bc72"}
Nov 03 20:08:23.615624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:23.611308    4421 scope.go:122] "RemoveContainer" containerID="70770412cdcc264a5b2066534ad21ca6417fb2c217de25164e9a7f7b28a3d8de"
Nov 03 20:08:23.615624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:23.611420    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d77cc995-4992w"
Nov 03 20:08:23.618896 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:23.618857    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-8lzxb" event={"ID":"c23a53b9-ca79-461a-a8f6-20618ffc35aa","Type":"ContainerStarted","Data":"9900e41d4e52b16eadb1134d47a689d09c247fbd720750ae814a0ec38a3aae55"}
Nov 03 20:08:23.619465 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:23.619439    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-8lzxb"
Nov 03 20:08:23.639563 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:23.639543    4421 scope.go:122] "RemoveContainer" containerID="70770412cdcc264a5b2066534ad21ca6417fb2c217de25164e9a7f7b28a3d8de"
Nov 03 20:08:23.640177 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:08:23.640109    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70770412cdcc264a5b2066534ad21ca6417fb2c217de25164e9a7f7b28a3d8de\": container with ID starting with 70770412cdcc264a5b2066534ad21ca6417fb2c217de25164e9a7f7b28a3d8de not found: ID does not exist" containerID="70770412cdcc264a5b2066534ad21ca6417fb2c217de25164e9a7f7b28a3d8de"
Nov 03 20:08:23.640177 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:23.640149    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70770412cdcc264a5b2066534ad21ca6417fb2c217de25164e9a7f7b28a3d8de"} err="failed to get container status \"70770412cdcc264a5b2066534ad21ca6417fb2c217de25164e9a7f7b28a3d8de\": rpc error: code = NotFound desc = could not find container \"70770412cdcc264a5b2066534ad21ca6417fb2c217de25164e9a7f7b28a3d8de\": container with ID starting with 70770412cdcc264a5b2066534ad21ca6417fb2c217de25164e9a7f7b28a3d8de not found: ID does not exist"
Nov 03 20:08:23.642917 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:23.642868    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-8lzxb" podStartSLOduration=2.318407284 podStartE2EDuration="5.642850586s" totalImagesPullingTime="3.324443302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:23.640835373 +0000 UTC m=+433.773963845" watchObservedRunningTime="2026-11-03 20:08:23.642850586 +0000 UTC m=+433.775979049"
Nov 03 20:08:23.655214 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:23.655186    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d77cc995-4992w"]
Nov 03 20:08:23.660455 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:23.660431    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5d77cc995-4992w"]
Nov 03 20:08:24.014527 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:24.014496    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12e64430-b367-4411-b366-d7c0f6e19eaf" path="/var/lib/kubelet/pods/12e64430-b367-4411-b366-d7c0f6e19eaf/volumes"
Nov 03 20:08:24.015021 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:24.014996    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b449554-2b61-4b92-91bc-dd4531ba3d8e" path="/var/lib/kubelet/pods/4b449554-2b61-4b92-91bc-dd4531ba3d8e/volumes"
Nov 03 20:08:24.629063 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:24.629026    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-2725f" event={"ID":"1c4bb3bf-caf2-4932-8b65-00189e8d41fb","Type":"ContainerStarted","Data":"ceb4326f1cb6aac88972eeef948d327cc1bb29f0920c5d52593e3497a65ccbb7"}
Nov 03 20:08:24.629501 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:24.629330    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-2725f"
Nov 03 20:08:24.649757 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:24.649696    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-2725f" podStartSLOduration=2.406989045 podStartE2EDuration="6.649677701s" totalImagesPullingTime="4.242688656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:24.648027435 +0000 UTC m=+434.781155898" watchObservedRunningTime="2026-11-03 20:08:24.649677701 +0000 UTC m=+434.782806166"
Nov 03 20:08:25.636136 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:25.636094    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-99ntv" event={"ID":"b347c3ff-a128-4fd2-8c07-dfea5c60a431","Type":"ContainerStarted","Data":"5e43d20582545c0105e529f431e5fb2e666478c9a29ba00bd82261f59649b5dc"}
Nov 03 20:08:25.656331 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:25.656273    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-99ntv" podStartSLOduration=2.366821544 podStartE2EDuration="7.656253838s" totalImagesPullingTime="5.289432294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:25.655352478 +0000 UTC m=+435.788480940" watchObservedRunningTime="2026-11-03 20:08:25.656253838 +0000 UTC m=+435.789382301"
Nov 03 20:08:26.641038 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:26.641006    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-99ntv"
Nov 03 20:08:30.660905 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:30.660868    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4bbvd" event={"ID":"1dcbd7f2-5145-444c-9b8f-7035eb14b64e","Type":"ContainerStarted","Data":"95eca98221a503abd18320fdea03edc249f793ae6156b3dabade2717114a50dd"}
Nov 03 20:08:30.661368 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:30.661031    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4bbvd"
Nov 03 20:08:30.681609 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:30.681542    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4bbvd" podStartSLOduration=1.757852484 podStartE2EDuration="18.68153104s" totalImagesPullingTime="16.923678556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:30.679790507 +0000 UTC m=+440.812918983" watchObservedRunningTime="2026-11-03 20:08:30.68153104 +0000 UTC m=+440.814659498"
Nov 03 20:08:35.641579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:35.641546    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-8lzxb"
Nov 03 20:08:37.648124 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:37.648095    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-99ntv"
Nov 03 20:08:41.670253 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:41.670222    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4bbvd"
Nov 03 20:08:49.627091 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:08:49.627048    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:08:55.560133 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.560099    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6474999cb4-52z85"]
Nov 03 20:08:55.560488 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.560435    4421 cpu_manager.go:414] "containerMap: removing container" podUID="4b449554-2b61-4b92-91bc-dd4531ba3d8e" containerName="manager"
Nov 03 20:08:55.560488 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.560445    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b449554-2b61-4b92-91bc-dd4531ba3d8e" containerName="manager"
Nov 03 20:08:55.560488 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.560459    4421 cpu_manager.go:414] "containerMap: removing container" podUID="12e64430-b367-4411-b366-d7c0f6e19eaf" containerName="discovery"
Nov 03 20:08:55.560488 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.560464    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12e64430-b367-4411-b366-d7c0f6e19eaf" containerName="discovery"
Nov 03 20:08:55.560637 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.560558    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b449554-2b61-4b92-91bc-dd4531ba3d8e" containerName="manager"
Nov 03 20:08:55.560637 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.560567    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="12e64430-b367-4411-b366-d7c0f6e19eaf" containerName="discovery"
Nov 03 20:08:55.562559 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.562540    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6474999cb4-52z85"
Nov 03 20:08:55.567349 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.567326    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:08:55.567349 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.567338    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:08:55.575873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.575851    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6474999cb4-52z85"]
Nov 03 20:08:55.607811 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.607773    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 20:08:55.607811 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.607781    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qqpwn\""
Nov 03 20:08:55.607968 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.607821    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 20:08:55.632243 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.632214    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0e46ea6-aded-4950-9550-3c35af396845-metrics-certs\") pod \"maas-api-6474999cb4-52z85\" (UID: \"e0e46ea6-aded-4950-9550-3c35af396845\") " pod="odh-ai-gateway-infra/maas-api-6474999cb4-52z85"
Nov 03 20:08:55.632335 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.632267    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e46ea6-aded-4950-9550-3c35af396845-openshift-service-ca\") pod \"maas-api-6474999cb4-52z85\" (UID: \"e0e46ea6-aded-4950-9550-3c35af396845\") " pod="odh-ai-gateway-infra/maas-api-6474999cb4-52z85"
Nov 03 20:08:55.632335 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.632298    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6grdx\" (UniqueName: \"kubernetes.io/projected/e0e46ea6-aded-4950-9550-3c35af396845-kube-api-access-6grdx\") pod \"maas-api-6474999cb4-52z85\" (UID: \"e0e46ea6-aded-4950-9550-3c35af396845\") " pod="odh-ai-gateway-infra/maas-api-6474999cb4-52z85"
Nov 03 20:08:55.632421 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.632372    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e46ea6-aded-4950-9550-3c35af396845-maas-api-tls\") pod \"maas-api-6474999cb4-52z85\" (UID: \"e0e46ea6-aded-4950-9550-3c35af396845\") " pod="odh-ai-gateway-infra/maas-api-6474999cb4-52z85"
Nov 03 20:08:55.642448 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.642425    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-2725f"
Nov 03 20:08:55.733323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.733296    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0e46ea6-aded-4950-9550-3c35af396845-metrics-certs\") pod \"maas-api-6474999cb4-52z85\" (UID: \"e0e46ea6-aded-4950-9550-3c35af396845\") " pod="odh-ai-gateway-infra/maas-api-6474999cb4-52z85"
Nov 03 20:08:55.733499 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.733341    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e46ea6-aded-4950-9550-3c35af396845-openshift-service-ca\") pod \"maas-api-6474999cb4-52z85\" (UID: \"e0e46ea6-aded-4950-9550-3c35af396845\") " pod="odh-ai-gateway-infra/maas-api-6474999cb4-52z85"
Nov 03 20:08:55.733499 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.733388    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6grdx\" (UniqueName: \"kubernetes.io/projected/e0e46ea6-aded-4950-9550-3c35af396845-kube-api-access-6grdx\") pod \"maas-api-6474999cb4-52z85\" (UID: \"e0e46ea6-aded-4950-9550-3c35af396845\") " pod="odh-ai-gateway-infra/maas-api-6474999cb4-52z85"
Nov 03 20:08:55.733499 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.733430    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e46ea6-aded-4950-9550-3c35af396845-maas-api-tls\") pod \"maas-api-6474999cb4-52z85\" (UID: \"e0e46ea6-aded-4950-9550-3c35af396845\") " pod="odh-ai-gateway-infra/maas-api-6474999cb4-52z85"
Nov 03 20:08:55.734060 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.734040    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e46ea6-aded-4950-9550-3c35af396845-openshift-service-ca\") pod \"maas-api-6474999cb4-52z85\" (UID: \"e0e46ea6-aded-4950-9550-3c35af396845\") " pod="odh-ai-gateway-infra/maas-api-6474999cb4-52z85"
Nov 03 20:08:55.735853 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.735828    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e46ea6-aded-4950-9550-3c35af396845-maas-api-tls\") pod \"maas-api-6474999cb4-52z85\" (UID: \"e0e46ea6-aded-4950-9550-3c35af396845\") " pod="odh-ai-gateway-infra/maas-api-6474999cb4-52z85"
Nov 03 20:08:55.735969 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.735927    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0e46ea6-aded-4950-9550-3c35af396845-metrics-certs\") pod \"maas-api-6474999cb4-52z85\" (UID: \"e0e46ea6-aded-4950-9550-3c35af396845\") " pod="odh-ai-gateway-infra/maas-api-6474999cb4-52z85"
Nov 03 20:08:55.748058 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.748033    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6grdx\" (UniqueName: \"kubernetes.io/projected/e0e46ea6-aded-4950-9550-3c35af396845-kube-api-access-6grdx\") pod \"maas-api-6474999cb4-52z85\" (UID: \"e0e46ea6-aded-4950-9550-3c35af396845\") " pod="odh-ai-gateway-infra/maas-api-6474999cb4-52z85"
Nov 03 20:08:55.771959 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.771933    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7f8b96f5f6-jjp75"]
Nov 03 20:08:55.774324 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.774306    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f8b96f5f6-jjp75"
Nov 03 20:08:55.779546 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.779530    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 03 20:08:55.783496 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.783472    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f8b96f5f6-jjp75"]
Nov 03 20:08:55.834196 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.834114    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsfgv\" (UniqueName: \"kubernetes.io/projected/2d1293d5-41dc-4c38-95c9-4af9b577705c-kube-api-access-dsfgv\") pod \"authorino-7f8b96f5f6-jjp75\" (UID: \"2d1293d5-41dc-4c38-95c9-4af9b577705c\") " pod="kuadrant-system/authorino-7f8b96f5f6-jjp75"
Nov 03 20:08:55.834196 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.834158    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2d1293d5-41dc-4c38-95c9-4af9b577705c-tls-cert\") pod \"authorino-7f8b96f5f6-jjp75\" (UID: \"2d1293d5-41dc-4c38-95c9-4af9b577705c\") " pod="kuadrant-system/authorino-7f8b96f5f6-jjp75"
Nov 03 20:08:55.873371 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.873340    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6474999cb4-52z85"
Nov 03 20:08:55.936705 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.935818    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2d1293d5-41dc-4c38-95c9-4af9b577705c-tls-cert\") pod \"authorino-7f8b96f5f6-jjp75\" (UID: \"2d1293d5-41dc-4c38-95c9-4af9b577705c\") " pod="kuadrant-system/authorino-7f8b96f5f6-jjp75"
Nov 03 20:08:55.936705 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.935981    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsfgv\" (UniqueName: \"kubernetes.io/projected/2d1293d5-41dc-4c38-95c9-4af9b577705c-kube-api-access-dsfgv\") pod \"authorino-7f8b96f5f6-jjp75\" (UID: \"2d1293d5-41dc-4c38-95c9-4af9b577705c\") " pod="kuadrant-system/authorino-7f8b96f5f6-jjp75"
Nov 03 20:08:55.948516 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.944260    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2d1293d5-41dc-4c38-95c9-4af9b577705c-tls-cert\") pod \"authorino-7f8b96f5f6-jjp75\" (UID: \"2d1293d5-41dc-4c38-95c9-4af9b577705c\") " pod="kuadrant-system/authorino-7f8b96f5f6-jjp75"
Nov 03 20:08:55.955377 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:55.955327    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsfgv\" (UniqueName: \"kubernetes.io/projected/2d1293d5-41dc-4c38-95c9-4af9b577705c-kube-api-access-dsfgv\") pod \"authorino-7f8b96f5f6-jjp75\" (UID: \"2d1293d5-41dc-4c38-95c9-4af9b577705c\") " pod="kuadrant-system/authorino-7f8b96f5f6-jjp75"
Nov 03 20:08:56.085996 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:56.085917    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f8b96f5f6-jjp75"
Nov 03 20:08:56.151587 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:56.151562    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6474999cb4-52z85"]
Nov 03 20:08:56.153344 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:08:56.153318    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e46ea6_aded_4950_9550_3c35af396845.slice/crio-730a11993a8bf9efd89b58309a177f219ffe3190619affb952529b69383d986d WatchSource:0}: Error finding container 730a11993a8bf9efd89b58309a177f219ffe3190619affb952529b69383d986d: Status 404 returned error can't find the container with id 730a11993a8bf9efd89b58309a177f219ffe3190619affb952529b69383d986d
Nov 03 20:08:56.354976 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:56.354956    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f8b96f5f6-jjp75"]
Nov 03 20:08:56.357625 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:08:56.357590    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d1293d5_41dc_4c38_95c9_4af9b577705c.slice/crio-e05f1921b0af490be014ad80b1bc2997d1239fb10b79aa9febb8ed9ffd992143 WatchSource:0}: Error finding container e05f1921b0af490be014ad80b1bc2997d1239fb10b79aa9febb8ed9ffd992143: Status 404 returned error can't find the container with id e05f1921b0af490be014ad80b1bc2997d1239fb10b79aa9febb8ed9ffd992143
Nov 03 20:08:56.768243 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:56.768207    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6474999cb4-52z85" event={"ID":"e0e46ea6-aded-4950-9550-3c35af396845","Type":"ContainerStarted","Data":"730a11993a8bf9efd89b58309a177f219ffe3190619affb952529b69383d986d"}
Nov 03 20:08:56.769513 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:56.769491    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f8b96f5f6-jjp75" event={"ID":"2d1293d5-41dc-4c38-95c9-4af9b577705c","Type":"ContainerStarted","Data":"e05f1921b0af490be014ad80b1bc2997d1239fb10b79aa9febb8ed9ffd992143"}
Nov 03 20:08:57.777967 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:57.777933    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f8b96f5f6-jjp75" event={"ID":"2d1293d5-41dc-4c38-95c9-4af9b577705c","Type":"ContainerStarted","Data":"aeb41215d47ded1d11967de9d9843be0dbb4333fbf68bf9faf712b13680eed18"}
Nov 03 20:08:57.824702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:57.822523    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7f8b96f5f6-jjp75" podStartSLOduration=1.887205628 podStartE2EDuration="2.822500982s" totalImagesPullingTime="935.295354ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:57.79686124 +0000 UTC m=+467.929989702" watchObservedRunningTime="2026-11-03 20:08:57.822500982 +0000 UTC m=+467.955629445"
Nov 03 20:08:57.824702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:57.823914    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-n49pr"]
Nov 03 20:08:57.824702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:57.824117    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-n49pr" podUID="e739b617-6018-48c9-9203-8f19a1213993" containerName="authorino" containerID="cri-o://fdcaf9e8dd89863b25e421f8eb756bc082b1f155f2c0f3a521b64a7583b2af4e" gracePeriod=30
Nov 03 20:08:58.110567 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:58.110533    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-n49pr"
Nov 03 20:08:58.262573 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:58.262548    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e739b617-6018-48c9-9203-8f19a1213993-kube-api-access-dfffw\" (UniqueName: \"kubernetes.io/projected/e739b617-6018-48c9-9203-8f19a1213993-kube-api-access-dfffw\") pod \"e739b617-6018-48c9-9203-8f19a1213993\" (UID: \"e739b617-6018-48c9-9203-8f19a1213993\") "
Nov 03 20:08:58.264684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:58.264661    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e739b617-6018-48c9-9203-8f19a1213993-kube-api-access-dfffw" pod "e739b617-6018-48c9-9203-8f19a1213993" (UID: "e739b617-6018-48c9-9203-8f19a1213993"). InnerVolumeSpecName "kube-api-access-dfffw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:58.364220 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:58.364154    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfffw\" (UniqueName: \"kubernetes.io/projected/e739b617-6018-48c9-9203-8f19a1213993-kube-api-access-dfffw\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:58.783818 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:58.783785    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6474999cb4-52z85" event={"ID":"e0e46ea6-aded-4950-9550-3c35af396845","Type":"ContainerStarted","Data":"15d2013e339e3a3bdebbb8424a0cc090856009e07be63dc2bc94e6aa623e85f7"}
Nov 03 20:08:58.784362 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:58.783921    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6474999cb4-52z85"
Nov 03 20:08:58.785099 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:58.785076    4421 generic.go:356] "Generic (PLEG): container finished" podID="e739b617-6018-48c9-9203-8f19a1213993" containerID="fdcaf9e8dd89863b25e421f8eb756bc082b1f155f2c0f3a521b64a7583b2af4e" exitCode=0
Nov 03 20:08:58.785220 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:58.785115    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-n49pr"
Nov 03 20:08:58.785220 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:58.785152    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-n49pr" event={"ID":"e739b617-6018-48c9-9203-8f19a1213993","Type":"ContainerDied","Data":"fdcaf9e8dd89863b25e421f8eb756bc082b1f155f2c0f3a521b64a7583b2af4e"}
Nov 03 20:08:58.785220 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:58.785184    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-n49pr" event={"ID":"e739b617-6018-48c9-9203-8f19a1213993","Type":"ContainerDied","Data":"91572eb684898f1cf4671b22efd0afa6fdf204f5f6a30b61d2741768d56ac0a4"}
Nov 03 20:08:58.785220 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:58.785200    4421 scope.go:122] "RemoveContainer" containerID="fdcaf9e8dd89863b25e421f8eb756bc082b1f155f2c0f3a521b64a7583b2af4e"
Nov 03 20:08:58.794258 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:58.794244    4421 scope.go:122] "RemoveContainer" containerID="fdcaf9e8dd89863b25e421f8eb756bc082b1f155f2c0f3a521b64a7583b2af4e"
Nov 03 20:08:58.794507 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:08:58.794489    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdcaf9e8dd89863b25e421f8eb756bc082b1f155f2c0f3a521b64a7583b2af4e\": container with ID starting with fdcaf9e8dd89863b25e421f8eb756bc082b1f155f2c0f3a521b64a7583b2af4e not found: ID does not exist" containerID="fdcaf9e8dd89863b25e421f8eb756bc082b1f155f2c0f3a521b64a7583b2af4e"
Nov 03 20:08:58.794570 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:58.794516    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdcaf9e8dd89863b25e421f8eb756bc082b1f155f2c0f3a521b64a7583b2af4e"} err="failed to get container status \"fdcaf9e8dd89863b25e421f8eb756bc082b1f155f2c0f3a521b64a7583b2af4e\": rpc error: code = NotFound desc = could not find container \"fdcaf9e8dd89863b25e421f8eb756bc082b1f155f2c0f3a521b64a7583b2af4e\": container with ID starting with fdcaf9e8dd89863b25e421f8eb756bc082b1f155f2c0f3a521b64a7583b2af4e not found: ID does not exist"
Nov 03 20:08:58.806136 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:58.806093    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6474999cb4-52z85" podStartSLOduration=1.719210406 podStartE2EDuration="3.806081529s" totalImagesPullingTime="2.086871123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:58.805076461 +0000 UTC m=+468.938204926" watchObservedRunningTime="2026-11-03 20:08:58.806081529 +0000 UTC m=+468.939209988"
Nov 03 20:08:58.818221 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:58.818200    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-n49pr"]
Nov 03 20:08:58.822026 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:08:58.822009    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-n49pr"]
Nov 03 20:09:00.014220 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:09:00.014188    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e739b617-6018-48c9-9203-8f19a1213993" path="/var/lib/kubelet/pods/e739b617-6018-48c9-9203-8f19a1213993/volumes"
Nov 03 20:09:04.795280 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:09:04.795254    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6474999cb4-52z85"
Nov 03 20:09:19.627909 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:09:19.627826    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:09:49.625761 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:09:49.625717    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:10:14.764318 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:14.764286    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-xx9xb"]
Nov 03 20:10:14.764804 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:14.764629    4421 cpu_manager.go:414] "containerMap: removing container" podUID="e739b617-6018-48c9-9203-8f19a1213993" containerName="authorino"
Nov 03 20:10:14.764804 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:14.764642    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e739b617-6018-48c9-9203-8f19a1213993" containerName="authorino"
Nov 03 20:10:14.764804 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:14.764725    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e739b617-6018-48c9-9203-8f19a1213993" containerName="authorino"
Nov 03 20:10:14.767644 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:14.767625    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-xx9xb"
Nov 03 20:10:14.775317 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:14.775296    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-76526\""
Nov 03 20:10:14.782111 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:14.782090    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-xx9xb"]
Nov 03 20:10:14.790032 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:14.790005    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79a63610-3e85-4ddd-a7a8-6ceccc2b3f83-plugins-config-volume\") pod \"payload-processing-7bb9489676-xx9xb\" (UID: \"79a63610-3e85-4ddd-a7a8-6ceccc2b3f83\") " pod="openshift-ingress/payload-processing-7bb9489676-xx9xb"
Nov 03 20:10:14.790155 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:14.790064    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smxs8\" (UniqueName: \"kubernetes.io/projected/79a63610-3e85-4ddd-a7a8-6ceccc2b3f83-kube-api-access-smxs8\") pod \"payload-processing-7bb9489676-xx9xb\" (UID: \"79a63610-3e85-4ddd-a7a8-6ceccc2b3f83\") " pod="openshift-ingress/payload-processing-7bb9489676-xx9xb"
Nov 03 20:10:14.860013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:14.859986    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 03 20:10:14.891461 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:14.891420    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smxs8\" (UniqueName: \"kubernetes.io/projected/79a63610-3e85-4ddd-a7a8-6ceccc2b3f83-kube-api-access-smxs8\") pod \"payload-processing-7bb9489676-xx9xb\" (UID: \"79a63610-3e85-4ddd-a7a8-6ceccc2b3f83\") " pod="openshift-ingress/payload-processing-7bb9489676-xx9xb"
Nov 03 20:10:14.891657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:14.891521    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79a63610-3e85-4ddd-a7a8-6ceccc2b3f83-plugins-config-volume\") pod \"payload-processing-7bb9489676-xx9xb\" (UID: \"79a63610-3e85-4ddd-a7a8-6ceccc2b3f83\") " pod="openshift-ingress/payload-processing-7bb9489676-xx9xb"
Nov 03 20:10:14.892267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:14.892242    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79a63610-3e85-4ddd-a7a8-6ceccc2b3f83-plugins-config-volume\") pod \"payload-processing-7bb9489676-xx9xb\" (UID: \"79a63610-3e85-4ddd-a7a8-6ceccc2b3f83\") " pod="openshift-ingress/payload-processing-7bb9489676-xx9xb"
Nov 03 20:10:14.903642 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:14.903614    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smxs8\" (UniqueName: \"kubernetes.io/projected/79a63610-3e85-4ddd-a7a8-6ceccc2b3f83-kube-api-access-smxs8\") pod \"payload-processing-7bb9489676-xx9xb\" (UID: \"79a63610-3e85-4ddd-a7a8-6ceccc2b3f83\") " pod="openshift-ingress/payload-processing-7bb9489676-xx9xb"
Nov 03 20:10:15.081007 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:15.080931    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-xx9xb"
Nov 03 20:10:15.354974 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:15.354952    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-xx9xb"]
Nov 03 20:10:15.358101 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:10:15.358066    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a63610_3e85_4ddd_a7a8_6ceccc2b3f83.slice/crio-e71c87a9998d5c082e7ede6d6cee018ff46039852d062b3c9437f19f8fa48f99 WatchSource:0}: Error finding container e71c87a9998d5c082e7ede6d6cee018ff46039852d062b3c9437f19f8fa48f99: Status 404 returned error can't find the container with id e71c87a9998d5c082e7ede6d6cee018ff46039852d062b3c9437f19f8fa48f99
Nov 03 20:10:16.091038 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:16.091000    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-xx9xb" event={"ID":"79a63610-3e85-4ddd-a7a8-6ceccc2b3f83","Type":"ContainerStarted","Data":"e71c87a9998d5c082e7ede6d6cee018ff46039852d062b3c9437f19f8fa48f99"}
Nov 03 20:10:18.101321 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:18.101283    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-xx9xb" event={"ID":"79a63610-3e85-4ddd-a7a8-6ceccc2b3f83","Type":"ContainerStarted","Data":"a782baadb7c5adbcf90db6c6c017abe4da2c8fb4d6762b2af08a3d5bb4be49d6"}
Nov 03 20:10:18.101729 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:18.101362    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7bb9489676-xx9xb"
Nov 03 20:10:18.120479 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:18.120430    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7bb9489676-xx9xb" podStartSLOduration=2.02089619 podStartE2EDuration="4.120419059s" totalImagesPullingTime="2.099522869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:18.118985944 +0000 UTC m=+548.252114400" watchObservedRunningTime="2026-11-03 20:10:18.120419059 +0000 UTC m=+548.253547520"
Nov 03 20:10:19.627127 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:10:19.627095    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:10:29.108040 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:29.108014    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7bb9489676-xx9xb"
Nov 03 20:10:47.410874 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:47.410840    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj"]
Nov 03 20:10:47.414718 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:47.414696    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj"
Nov 03 20:10:47.424771 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:47.424747    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vmkc9\""
Nov 03 20:10:47.424949 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:47.424899    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 03 20:10:47.425086 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:47.424990    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 20:10:47.425143 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:47.425087    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 20:10:47.425514 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:47.425494    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj"]
Nov 03 20:10:47.459615 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:47.459566    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz5s7\" (UniqueName: \"kubernetes.io/projected/40f4d063-2060-49f8-8b17-13526284ed89-kube-api-access-lz5s7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj\" (UID: \"40f4d063-2060-49f8-8b17-13526284ed89\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj"
Nov 03 20:10:47.459764 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:47.459684    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40f4d063-2060-49f8-8b17-13526284ed89-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj\" (UID: \"40f4d063-2060-49f8-8b17-13526284ed89\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj"
Nov 03 20:10:47.459764 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:47.459722    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40f4d063-2060-49f8-8b17-13526284ed89-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj\" (UID: \"40f4d063-2060-49f8-8b17-13526284ed89\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj"
Nov 03 20:10:47.459764 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:47.459748    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40f4d063-2060-49f8-8b17-13526284ed89-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj\" (UID: \"40f4d063-2060-49f8-8b17-13526284ed89\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj"
Nov 03 20:10:47.459880 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:47.459790    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40f4d063-2060-49f8-8b17-13526284ed89-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj\" (UID: \"40f4d063-2060-49f8-8b17-13526284ed89\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj"
Nov 03 20:10:47.459880 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:47.459823    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40f4d063-2060-49f8-8b17-13526284ed89-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj\" (UID: \"40f4d063-2060-49f8-8b17-13526284ed89\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj"
Nov 03 20:10:47.560774 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:47.560740    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40f4d063-2060-49f8-8b17-13526284ed89-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj\" (UID: \"40f4d063-2060-49f8-8b17-13526284ed89\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj"
Nov 03 20:10:47.560944 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:47.560792    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz5s7\" (UniqueName: \"kubernetes.io/projected/40f4d063-2060-49f8-8b17-13526284ed89-kube-api-access-lz5s7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj\" (UID: \"40f4d063-2060-49f8-8b17-13526284ed89\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj"
Nov 03 20:10:47.560944 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:47.560835    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40f4d063-2060-49f8-8b17-13526284ed89-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj\" (UID: \"40f4d063-2060-49f8-8b17-13526284ed89\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj"
Nov 03 20:10:47.560944 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:47.560856    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40f4d063-2060-49f8-8b17-13526284ed89-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj\" (UID: \"40f4d063-2060-49f8-8b17-13526284ed89\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj"
Nov 03 20:10:47.560944 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:47.560875    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40f4d063-2060-49f8-8b17-13526284ed89-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj\" (UID: \"40f4d063-2060-49f8-8b17-13526284ed89\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj"
Nov 03 20:10:47.560944 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:47.560889    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40f4d063-2060-49f8-8b17-13526284ed89-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj\" (UID: \"40f4d063-2060-49f8-8b17-13526284ed89\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj"
Nov 03 20:10:47.561208 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:47.561163    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40f4d063-2060-49f8-8b17-13526284ed89-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj\" (UID: \"40f4d063-2060-49f8-8b17-13526284ed89\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj"
Nov 03 20:10:47.561253 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:47.561234    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40f4d063-2060-49f8-8b17-13526284ed89-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj\" (UID: \"40f4d063-2060-49f8-8b17-13526284ed89\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj"
Nov 03 20:10:47.561363 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:47.561324    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40f4d063-2060-49f8-8b17-13526284ed89-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj\" (UID: \"40f4d063-2060-49f8-8b17-13526284ed89\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj"
Nov 03 20:10:47.563360 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:47.563338    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40f4d063-2060-49f8-8b17-13526284ed89-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj\" (UID: \"40f4d063-2060-49f8-8b17-13526284ed89\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj"
Nov 03 20:10:47.563900 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:47.563879    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40f4d063-2060-49f8-8b17-13526284ed89-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj\" (UID: \"40f4d063-2060-49f8-8b17-13526284ed89\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj"
Nov 03 20:10:47.573303 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:47.573282    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz5s7\" (UniqueName: \"kubernetes.io/projected/40f4d063-2060-49f8-8b17-13526284ed89-kube-api-access-lz5s7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj\" (UID: \"40f4d063-2060-49f8-8b17-13526284ed89\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj"
Nov 03 20:10:47.730585 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:47.730548    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj"
Nov 03 20:10:48.019266 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:48.019185    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj"]
Nov 03 20:10:48.219420 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:48.219384    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj" event={"ID":"40f4d063-2060-49f8-8b17-13526284ed89","Type":"ContainerStarted","Data":"ee998dd0955c2e6d6ca179e4f099345ba0412121ea6fb502c9f667112e068bc9"}
Nov 03 20:10:49.627435 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:10:49.627287    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:10:50.228688 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:50.228645    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj" event={"ID":"40f4d063-2060-49f8-8b17-13526284ed89","Type":"ContainerStarted","Data":"1f0ec0ad94271b44ef2059bfe9b69c6f4c031c253c23fdac630871365ae3a5a7"}
Nov 03 20:10:50.251313 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:50.251247    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj" podStartSLOduration=2.002271678 podStartE2EDuration="3.251195963s" totalImagesPullingTime="1.248924285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:50.248992679 +0000 UTC m=+580.382121141" watchObservedRunningTime="2026-11-03 20:10:50.251195963 +0000 UTC m=+580.384324426"
Nov 03 20:10:50.730944 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:50.730911    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj"
Nov 03 20:10:53.727868 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:53.727821    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8"]
Nov 03 20:10:53.731136 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:53.731113    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8"
Nov 03 20:10:53.739194 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:53.739174    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-external-token-limit-simulated-kserve-self-signed-certs\""
Nov 03 20:10:53.743945 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:53.743921    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8"]
Nov 03 20:10:53.818733 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:53.818700    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d37ba58a-679f-4489-ab67-001ea89cf66d-dshm\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8\" (UID: \"d37ba58a-679f-4489-ab67-001ea89cf66d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8"
Nov 03 20:10:53.818883 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:53.818746    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tj4q\" (UniqueName: \"kubernetes.io/projected/d37ba58a-679f-4489-ab67-001ea89cf66d-kube-api-access-5tj4q\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8\" (UID: \"d37ba58a-679f-4489-ab67-001ea89cf66d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8"
Nov 03 20:10:53.818883 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:53.818842    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37ba58a-679f-4489-ab67-001ea89cf66d-model-cache\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8\" (UID: \"d37ba58a-679f-4489-ab67-001ea89cf66d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8"
Nov 03 20:10:53.818960 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:53.818903    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d37ba58a-679f-4489-ab67-001ea89cf66d-tmp-dir\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8\" (UID: \"d37ba58a-679f-4489-ab67-001ea89cf66d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8"
Nov 03 20:10:53.818960 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:53.818926    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d37ba58a-679f-4489-ab67-001ea89cf66d-home\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8\" (UID: \"d37ba58a-679f-4489-ab67-001ea89cf66d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8"
Nov 03 20:10:53.819025 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:53.818994    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37ba58a-679f-4489-ab67-001ea89cf66d-tls-certs\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8\" (UID: \"d37ba58a-679f-4489-ab67-001ea89cf66d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8"
Nov 03 20:10:53.920265 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:53.920237    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37ba58a-679f-4489-ab67-001ea89cf66d-model-cache\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8\" (UID: \"d37ba58a-679f-4489-ab67-001ea89cf66d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8"
Nov 03 20:10:53.920445 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:53.920296    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d37ba58a-679f-4489-ab67-001ea89cf66d-tmp-dir\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8\" (UID: \"d37ba58a-679f-4489-ab67-001ea89cf66d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8"
Nov 03 20:10:53.920445 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:53.920323    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d37ba58a-679f-4489-ab67-001ea89cf66d-home\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8\" (UID: \"d37ba58a-679f-4489-ab67-001ea89cf66d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8"
Nov 03 20:10:53.920445 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:53.920346    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37ba58a-679f-4489-ab67-001ea89cf66d-tls-certs\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8\" (UID: \"d37ba58a-679f-4489-ab67-001ea89cf66d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8"
Nov 03 20:10:53.920445 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:53.920364    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d37ba58a-679f-4489-ab67-001ea89cf66d-dshm\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8\" (UID: \"d37ba58a-679f-4489-ab67-001ea89cf66d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8"
Nov 03 20:10:53.920445 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:53.920385    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tj4q\" (UniqueName: \"kubernetes.io/projected/d37ba58a-679f-4489-ab67-001ea89cf66d-kube-api-access-5tj4q\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8\" (UID: \"d37ba58a-679f-4489-ab67-001ea89cf66d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8"
Nov 03 20:10:53.920760 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:53.920717    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d37ba58a-679f-4489-ab67-001ea89cf66d-tmp-dir\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8\" (UID: \"d37ba58a-679f-4489-ab67-001ea89cf66d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8"
Nov 03 20:10:53.920802 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:53.920770    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d37ba58a-679f-4489-ab67-001ea89cf66d-home\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8\" (UID: \"d37ba58a-679f-4489-ab67-001ea89cf66d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8"
Nov 03 20:10:53.920835 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:53.920818    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37ba58a-679f-4489-ab67-001ea89cf66d-model-cache\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8\" (UID: \"d37ba58a-679f-4489-ab67-001ea89cf66d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8"
Nov 03 20:10:53.922783 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:53.922759    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d37ba58a-679f-4489-ab67-001ea89cf66d-dshm\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8\" (UID: \"d37ba58a-679f-4489-ab67-001ea89cf66d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8"
Nov 03 20:10:53.922889 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:53.922873    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37ba58a-679f-4489-ab67-001ea89cf66d-tls-certs\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8\" (UID: \"d37ba58a-679f-4489-ab67-001ea89cf66d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8"
Nov 03 20:10:53.933312 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:53.933291    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tj4q\" (UniqueName: \"kubernetes.io/projected/d37ba58a-679f-4489-ab67-001ea89cf66d-kube-api-access-5tj4q\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8\" (UID: \"d37ba58a-679f-4489-ab67-001ea89cf66d\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8"
Nov 03 20:10:54.045830 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:54.045747    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8"
Nov 03 20:10:54.322253 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:54.322206    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8"]
Nov 03 20:10:54.324621 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:10:54.324574    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37ba58a_679f_4489_ab67_001ea89cf66d.slice/crio-cbcd83df10f48407241a0bae7e3fdf969b884933d2830467876014b88fd1b79b WatchSource:0}: Error finding container cbcd83df10f48407241a0bae7e3fdf969b884933d2830467876014b88fd1b79b: Status 404 returned error can't find the container with id cbcd83df10f48407241a0bae7e3fdf969b884933d2830467876014b88fd1b79b
Nov 03 20:10:55.250673 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:55.250639    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8" event={"ID":"d37ba58a-679f-4489-ab67-001ea89cf66d","Type":"ContainerStarted","Data":"03b4e82de68ec70001f3964b189215bc19845ce089fe113f853f412b2c762dc2"}
Nov 03 20:10:55.251175 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:55.250704    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8" event={"ID":"d37ba58a-679f-4489-ab67-001ea89cf66d","Type":"ContainerStarted","Data":"cbcd83df10f48407241a0bae7e3fdf969b884933d2830467876014b88fd1b79b"}
Nov 03 20:10:55.271895 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:55.271801    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8" podStartSLOduration=1.8786003070000001 podStartE2EDuration="2.271782408s" totalImagesPullingTime="393.182101ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:55.271087323 +0000 UTC m=+585.404215807" watchObservedRunningTime="2026-11-03 20:10:55.271782408 +0000 UTC m=+585.404910870"
Nov 03 20:10:56.046168 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:56.046136    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8"
Nov 03 20:10:56.286029 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:56.285981    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv"]
Nov 03 20:10:56.290643 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:56.290618    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv"
Nov 03 20:10:56.309064 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:56.309041    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 03 20:10:56.310639 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:56.310612    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv"]
Nov 03 20:10:56.443222 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:56.443185    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc9f7589-f4e9-4445-95cb-7b0cc446cee4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv\" (UID: \"cc9f7589-f4e9-4445-95cb-7b0cc446cee4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv"
Nov 03 20:10:56.443475 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:56.443253    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp87d\" (UniqueName: \"kubernetes.io/projected/cc9f7589-f4e9-4445-95cb-7b0cc446cee4-kube-api-access-sp87d\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv\" (UID: \"cc9f7589-f4e9-4445-95cb-7b0cc446cee4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv"
Nov 03 20:10:56.443475 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:56.443370    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc9f7589-f4e9-4445-95cb-7b0cc446cee4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv\" (UID: \"cc9f7589-f4e9-4445-95cb-7b0cc446cee4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv"
Nov 03 20:10:56.443475 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:56.443402    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc9f7589-f4e9-4445-95cb-7b0cc446cee4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv\" (UID: \"cc9f7589-f4e9-4445-95cb-7b0cc446cee4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv"
Nov 03 20:10:56.443475 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:56.443446    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc9f7589-f4e9-4445-95cb-7b0cc446cee4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv\" (UID: \"cc9f7589-f4e9-4445-95cb-7b0cc446cee4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv"
Nov 03 20:10:56.443711 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:56.443476    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc9f7589-f4e9-4445-95cb-7b0cc446cee4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv\" (UID: \"cc9f7589-f4e9-4445-95cb-7b0cc446cee4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv"
Nov 03 20:10:56.544260 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:56.544222    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp87d\" (UniqueName: \"kubernetes.io/projected/cc9f7589-f4e9-4445-95cb-7b0cc446cee4-kube-api-access-sp87d\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv\" (UID: \"cc9f7589-f4e9-4445-95cb-7b0cc446cee4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv"
Nov 03 20:10:56.544260 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:56.544266    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc9f7589-f4e9-4445-95cb-7b0cc446cee4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv\" (UID: \"cc9f7589-f4e9-4445-95cb-7b0cc446cee4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv"
Nov 03 20:10:56.544512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:56.544285    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc9f7589-f4e9-4445-95cb-7b0cc446cee4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv\" (UID: \"cc9f7589-f4e9-4445-95cb-7b0cc446cee4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv"
Nov 03 20:10:56.544512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:56.544351    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc9f7589-f4e9-4445-95cb-7b0cc446cee4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv\" (UID: \"cc9f7589-f4e9-4445-95cb-7b0cc446cee4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv"
Nov 03 20:10:56.544512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:56.544400    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc9f7589-f4e9-4445-95cb-7b0cc446cee4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv\" (UID: \"cc9f7589-f4e9-4445-95cb-7b0cc446cee4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv"
Nov 03 20:10:56.544512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:56.544493    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc9f7589-f4e9-4445-95cb-7b0cc446cee4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv\" (UID: \"cc9f7589-f4e9-4445-95cb-7b0cc446cee4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv"
Nov 03 20:10:56.544827 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:56.544804    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc9f7589-f4e9-4445-95cb-7b0cc446cee4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv\" (UID: \"cc9f7589-f4e9-4445-95cb-7b0cc446cee4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv"
Nov 03 20:10:56.544894 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:56.544838    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc9f7589-f4e9-4445-95cb-7b0cc446cee4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv\" (UID: \"cc9f7589-f4e9-4445-95cb-7b0cc446cee4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv"
Nov 03 20:10:56.544945 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:56.544886    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc9f7589-f4e9-4445-95cb-7b0cc446cee4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv\" (UID: \"cc9f7589-f4e9-4445-95cb-7b0cc446cee4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv"
Nov 03 20:10:56.546820 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:56.546725    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc9f7589-f4e9-4445-95cb-7b0cc446cee4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv\" (UID: \"cc9f7589-f4e9-4445-95cb-7b0cc446cee4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv"
Nov 03 20:10:56.547302 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:56.547281    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc9f7589-f4e9-4445-95cb-7b0cc446cee4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv\" (UID: \"cc9f7589-f4e9-4445-95cb-7b0cc446cee4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv"
Nov 03 20:10:56.559877 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:56.559823    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp87d\" (UniqueName: \"kubernetes.io/projected/cc9f7589-f4e9-4445-95cb-7b0cc446cee4-kube-api-access-sp87d\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv\" (UID: \"cc9f7589-f4e9-4445-95cb-7b0cc446cee4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv"
Nov 03 20:10:56.612915 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:56.612880    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv"
Nov 03 20:10:56.899499 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:56.899477    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv"]
Nov 03 20:10:56.901344 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:10:56.901307    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9f7589_f4e9_4445_95cb_7b0cc446cee4.slice/crio-c9b135aaaf4cfabc620a123563de8b8e9d037bb9f9f1a8341a52fb257eae9185 WatchSource:0}: Error finding container c9b135aaaf4cfabc620a123563de8b8e9d037bb9f9f1a8341a52fb257eae9185: Status 404 returned error can't find the container with id c9b135aaaf4cfabc620a123563de8b8e9d037bb9f9f1a8341a52fb257eae9185
Nov 03 20:10:57.259694 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:57.259666    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv" event={"ID":"cc9f7589-f4e9-4445-95cb-7b0cc446cee4","Type":"ContainerStarted","Data":"c9b135aaaf4cfabc620a123563de8b8e9d037bb9f9f1a8341a52fb257eae9185"}
Nov 03 20:10:57.731351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:57.731322    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj"
Nov 03 20:10:57.743577 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:57.743557    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj"
Nov 03 20:10:58.278142 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:58.278084    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv" event={"ID":"cc9f7589-f4e9-4445-95cb-7b0cc446cee4","Type":"ContainerStarted","Data":"526ace4e85fb78e52add01bc33ee264da78d74c3f93760d1602d70d21938ed02"}
Nov 03 20:10:58.290384 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:58.290354    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj"
Nov 03 20:10:58.300459 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:58.300414    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv" podStartSLOduration=1.952990517 podStartE2EDuration="2.300402042s" totalImagesPullingTime="347.411525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:58.297969154 +0000 UTC m=+588.431097616" watchObservedRunningTime="2026-11-03 20:10:58.300402042 +0000 UTC m=+588.433530503"
Nov 03 20:10:58.613686 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:58.613582    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv"
Nov 03 20:10:59.794297 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:59.794256    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs"]
Nov 03 20:10:59.798805 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:59.798780    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs"
Nov 03 20:10:59.808275 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:59.808249    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 03 20:10:59.814430 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:59.814406    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs"]
Nov 03 20:10:59.883133 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:59.883090    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e155365-719f-4ee2-bdd0-08b076f7513c-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs\" (UID: \"7e155365-719f-4ee2-bdd0-08b076f7513c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs"
Nov 03 20:10:59.883325 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:59.883140    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e155365-719f-4ee2-bdd0-08b076f7513c-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs\" (UID: \"7e155365-719f-4ee2-bdd0-08b076f7513c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs"
Nov 03 20:10:59.883325 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:59.883178    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e155365-719f-4ee2-bdd0-08b076f7513c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs\" (UID: \"7e155365-719f-4ee2-bdd0-08b076f7513c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs"
Nov 03 20:10:59.883325 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:59.883292    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e155365-719f-4ee2-bdd0-08b076f7513c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs\" (UID: \"7e155365-719f-4ee2-bdd0-08b076f7513c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs"
Nov 03 20:10:59.883463 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:59.883379    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e155365-719f-4ee2-bdd0-08b076f7513c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs\" (UID: \"7e155365-719f-4ee2-bdd0-08b076f7513c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs"
Nov 03 20:10:59.883463 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:59.883450    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbrgh\" (UniqueName: \"kubernetes.io/projected/7e155365-719f-4ee2-bdd0-08b076f7513c-kube-api-access-bbrgh\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs\" (UID: \"7e155365-719f-4ee2-bdd0-08b076f7513c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs"
Nov 03 20:10:59.984542 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:59.984510    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbrgh\" (UniqueName: \"kubernetes.io/projected/7e155365-719f-4ee2-bdd0-08b076f7513c-kube-api-access-bbrgh\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs\" (UID: \"7e155365-719f-4ee2-bdd0-08b076f7513c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs"
Nov 03 20:10:59.984724 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:59.984580    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e155365-719f-4ee2-bdd0-08b076f7513c-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs\" (UID: \"7e155365-719f-4ee2-bdd0-08b076f7513c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs"
Nov 03 20:10:59.984724 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:59.984613    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e155365-719f-4ee2-bdd0-08b076f7513c-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs\" (UID: \"7e155365-719f-4ee2-bdd0-08b076f7513c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs"
Nov 03 20:10:59.984724 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:59.984632    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e155365-719f-4ee2-bdd0-08b076f7513c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs\" (UID: \"7e155365-719f-4ee2-bdd0-08b076f7513c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs"
Nov 03 20:10:59.984724 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:59.984674    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e155365-719f-4ee2-bdd0-08b076f7513c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs\" (UID: \"7e155365-719f-4ee2-bdd0-08b076f7513c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs"
Nov 03 20:10:59.984724 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:59.984699    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e155365-719f-4ee2-bdd0-08b076f7513c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs\" (UID: \"7e155365-719f-4ee2-bdd0-08b076f7513c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs"
Nov 03 20:10:59.985100 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:59.985073    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e155365-719f-4ee2-bdd0-08b076f7513c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs\" (UID: \"7e155365-719f-4ee2-bdd0-08b076f7513c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs"
Nov 03 20:10:59.985239 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:59.985185    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e155365-719f-4ee2-bdd0-08b076f7513c-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs\" (UID: \"7e155365-719f-4ee2-bdd0-08b076f7513c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs"
Nov 03 20:10:59.985239 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:59.985193    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e155365-719f-4ee2-bdd0-08b076f7513c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs\" (UID: \"7e155365-719f-4ee2-bdd0-08b076f7513c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs"
Nov 03 20:10:59.987081 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:59.987060    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e155365-719f-4ee2-bdd0-08b076f7513c-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs\" (UID: \"7e155365-719f-4ee2-bdd0-08b076f7513c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs"
Nov 03 20:10:59.987419 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:10:59.987399    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e155365-719f-4ee2-bdd0-08b076f7513c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs\" (UID: \"7e155365-719f-4ee2-bdd0-08b076f7513c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs"
Nov 03 20:11:00.001344 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:00.001319    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbrgh\" (UniqueName: \"kubernetes.io/projected/7e155365-719f-4ee2-bdd0-08b076f7513c-kube-api-access-bbrgh\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs\" (UID: \"7e155365-719f-4ee2-bdd0-08b076f7513c\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs"
Nov 03 20:11:00.115161 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:00.115085    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs"
Nov 03 20:11:00.403438 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:00.403414    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs"]
Nov 03 20:11:00.406129 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:11:00.406097    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e155365_719f_4ee2_bdd0_08b076f7513c.slice/crio-737d33208c9c6238df6c6c1be626f6e49bc38169382d16d2d5ad42c47b8a15d3 WatchSource:0}: Error finding container 737d33208c9c6238df6c6c1be626f6e49bc38169382d16d2d5ad42c47b8a15d3: Status 404 returned error can't find the container with id 737d33208c9c6238df6c6c1be626f6e49bc38169382d16d2d5ad42c47b8a15d3
Nov 03 20:11:01.295138 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:01.295107    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs" event={"ID":"7e155365-719f-4ee2-bdd0-08b076f7513c","Type":"ContainerStarted","Data":"c94b3a5482f5824e38a6417101439de91ce66a497f0e76c91f5f2f97c329248c"}
Nov 03 20:11:01.295587 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:01.295154    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs" event={"ID":"7e155365-719f-4ee2-bdd0-08b076f7513c","Type":"ContainerStarted","Data":"737d33208c9c6238df6c6c1be626f6e49bc38169382d16d2d5ad42c47b8a15d3"}
Nov 03 20:11:01.316304 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:01.316259    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs" podStartSLOduration=1.9489516679999999 podStartE2EDuration="2.316245063s" totalImagesPullingTime="367.293395ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:01.314903945 +0000 UTC m=+591.448032406" watchObservedRunningTime="2026-11-03 20:11:01.316245063 +0000 UTC m=+591.449373583"
Nov 03 20:11:02.116043 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:02.116005    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs"
Nov 03 20:11:04.046877 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:04.046845    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8"
Nov 03 20:11:04.059279 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:04.059258    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8"
Nov 03 20:11:04.321228 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:04.321144    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8"
Nov 03 20:11:06.613293 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:06.613259    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv"
Nov 03 20:11:06.625655 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:06.625629    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv"
Nov 03 20:11:07.332616 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:07.332567    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv"
Nov 03 20:11:10.115306 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:10.115224    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs"
Nov 03 20:11:10.127851 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:10.127824    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs"
Nov 03 20:11:10.345619 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:10.345580    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs"
Nov 03 20:11:18.002080 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:18.002048    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cfb485dff-mjxfv"]
Nov 03 20:11:18.005730 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:18.005714    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cfb485dff-mjxfv"
Nov 03 20:11:18.015035 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:18.015009    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cfb485dff-mjxfv"]
Nov 03 20:11:18.144063 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:18.144025    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqtnv\" (UniqueName: \"kubernetes.io/projected/6f1f72d4-3cc6-4ae2-b9ac-a399a5b9a34c-kube-api-access-nqtnv\") pod \"authorino-7cfb485dff-mjxfv\" (UID: \"6f1f72d4-3cc6-4ae2-b9ac-a399a5b9a34c\") " pod="kuadrant-system/authorino-7cfb485dff-mjxfv"
Nov 03 20:11:18.144063 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:18.144077    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6f1f72d4-3cc6-4ae2-b9ac-a399a5b9a34c-tls-cert\") pod \"authorino-7cfb485dff-mjxfv\" (UID: \"6f1f72d4-3cc6-4ae2-b9ac-a399a5b9a34c\") " pod="kuadrant-system/authorino-7cfb485dff-mjxfv"
Nov 03 20:11:18.244769 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:18.244714    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqtnv\" (UniqueName: \"kubernetes.io/projected/6f1f72d4-3cc6-4ae2-b9ac-a399a5b9a34c-kube-api-access-nqtnv\") pod \"authorino-7cfb485dff-mjxfv\" (UID: \"6f1f72d4-3cc6-4ae2-b9ac-a399a5b9a34c\") " pod="kuadrant-system/authorino-7cfb485dff-mjxfv"
Nov 03 20:11:18.244769 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:18.244778    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6f1f72d4-3cc6-4ae2-b9ac-a399a5b9a34c-tls-cert\") pod \"authorino-7cfb485dff-mjxfv\" (UID: \"6f1f72d4-3cc6-4ae2-b9ac-a399a5b9a34c\") " pod="kuadrant-system/authorino-7cfb485dff-mjxfv"
Nov 03 20:11:18.247380 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:18.247358    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6f1f72d4-3cc6-4ae2-b9ac-a399a5b9a34c-tls-cert\") pod \"authorino-7cfb485dff-mjxfv\" (UID: \"6f1f72d4-3cc6-4ae2-b9ac-a399a5b9a34c\") " pod="kuadrant-system/authorino-7cfb485dff-mjxfv"
Nov 03 20:11:18.257548 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:18.257487    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqtnv\" (UniqueName: \"kubernetes.io/projected/6f1f72d4-3cc6-4ae2-b9ac-a399a5b9a34c-kube-api-access-nqtnv\") pod \"authorino-7cfb485dff-mjxfv\" (UID: \"6f1f72d4-3cc6-4ae2-b9ac-a399a5b9a34c\") " pod="kuadrant-system/authorino-7cfb485dff-mjxfv"
Nov 03 20:11:18.319874 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:18.319844    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cfb485dff-mjxfv"
Nov 03 20:11:18.592889 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:18.592860    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cfb485dff-mjxfv"]
Nov 03 20:11:18.594922 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:11:18.594893    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f1f72d4_3cc6_4ae2_b9ac_a399a5b9a34c.slice/crio-717fe6e1a579c733fcb9c66ab90392f55a2b134958e6df4818ea0cabdbe76748 WatchSource:0}: Error finding container 717fe6e1a579c733fcb9c66ab90392f55a2b134958e6df4818ea0cabdbe76748: Status 404 returned error can't find the container with id 717fe6e1a579c733fcb9c66ab90392f55a2b134958e6df4818ea0cabdbe76748
Nov 03 20:11:19.381273 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:19.381239    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cfb485dff-mjxfv" event={"ID":"6f1f72d4-3cc6-4ae2-b9ac-a399a5b9a34c","Type":"ContainerStarted","Data":"717fe6e1a579c733fcb9c66ab90392f55a2b134958e6df4818ea0cabdbe76748"}
Nov 03 20:11:19.627023 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:11:19.626989    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:11:20.387442 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:20.387409    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cfb485dff-mjxfv" event={"ID":"6f1f72d4-3cc6-4ae2-b9ac-a399a5b9a34c","Type":"ContainerStarted","Data":"7eb72091e9cf3b3b3e98a52b5531a1d3d9c26b9c49c91ac7109ac935e43308a4"}
Nov 03 20:11:20.409790 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:20.409735    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7cfb485dff-mjxfv" podStartSLOduration=2.283387414 podStartE2EDuration="3.409721098s" totalImagesPullingTime="1.126333684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:20.408416151 +0000 UTC m=+610.541544615" watchObservedRunningTime="2026-11-03 20:11:20.409721098 +0000 UTC m=+610.542849555"
Nov 03 20:11:20.439278 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:20.439243    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f8b96f5f6-jjp75"]
Nov 03 20:11:20.439495 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:20.439469    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7f8b96f5f6-jjp75" podUID="2d1293d5-41dc-4c38-95c9-4af9b577705c" containerName="authorino" containerID="cri-o://aeb41215d47ded1d11967de9d9843be0dbb4333fbf68bf9faf712b13680eed18" gracePeriod=30
Nov 03 20:11:20.708891 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:20.708867    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f8b96f5f6-jjp75"
Nov 03 20:11:20.766095 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:20.766067    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d1293d5-41dc-4c38-95c9-4af9b577705c-kube-api-access-dsfgv\" (UniqueName: \"kubernetes.io/projected/2d1293d5-41dc-4c38-95c9-4af9b577705c-kube-api-access-dsfgv\") pod \"2d1293d5-41dc-4c38-95c9-4af9b577705c\" (UID: \"2d1293d5-41dc-4c38-95c9-4af9b577705c\") "
Nov 03 20:11:20.766251 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:20.766158    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d1293d5-41dc-4c38-95c9-4af9b577705c-tls-cert\" (UniqueName: \"kubernetes.io/secret/2d1293d5-41dc-4c38-95c9-4af9b577705c-tls-cert\") pod \"2d1293d5-41dc-4c38-95c9-4af9b577705c\" (UID: \"2d1293d5-41dc-4c38-95c9-4af9b577705c\") "
Nov 03 20:11:20.768306 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:20.768269    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1293d5-41dc-4c38-95c9-4af9b577705c-kube-api-access-dsfgv" pod "2d1293d5-41dc-4c38-95c9-4af9b577705c" (UID: "2d1293d5-41dc-4c38-95c9-4af9b577705c"). InnerVolumeSpecName "kube-api-access-dsfgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:20.777030 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:20.777002    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d1293d5-41dc-4c38-95c9-4af9b577705c-tls-cert" pod "2d1293d5-41dc-4c38-95c9-4af9b577705c" (UID: "2d1293d5-41dc-4c38-95c9-4af9b577705c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:11:20.867174 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:20.867110    4421 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2d1293d5-41dc-4c38-95c9-4af9b577705c-tls-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:20.867174 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:20.867147    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsfgv\" (UniqueName: \"kubernetes.io/projected/2d1293d5-41dc-4c38-95c9-4af9b577705c-kube-api-access-dsfgv\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:21.393533 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:21.393499    4421 generic.go:356] "Generic (PLEG): container finished" podID="2d1293d5-41dc-4c38-95c9-4af9b577705c" containerID="aeb41215d47ded1d11967de9d9843be0dbb4333fbf68bf9faf712b13680eed18" exitCode=0
Nov 03 20:11:21.394058 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:21.393582    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f8b96f5f6-jjp75"
Nov 03 20:11:21.394058 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:21.393634    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f8b96f5f6-jjp75" event={"ID":"2d1293d5-41dc-4c38-95c9-4af9b577705c","Type":"ContainerDied","Data":"aeb41215d47ded1d11967de9d9843be0dbb4333fbf68bf9faf712b13680eed18"}
Nov 03 20:11:21.394058 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:21.393676    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f8b96f5f6-jjp75" event={"ID":"2d1293d5-41dc-4c38-95c9-4af9b577705c","Type":"ContainerDied","Data":"e05f1921b0af490be014ad80b1bc2997d1239fb10b79aa9febb8ed9ffd992143"}
Nov 03 20:11:21.394058 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:21.393698    4421 scope.go:122] "RemoveContainer" containerID="aeb41215d47ded1d11967de9d9843be0dbb4333fbf68bf9faf712b13680eed18"
Nov 03 20:11:21.402912 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:21.402894    4421 scope.go:122] "RemoveContainer" containerID="aeb41215d47ded1d11967de9d9843be0dbb4333fbf68bf9faf712b13680eed18"
Nov 03 20:11:21.403198 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:11:21.403179    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeb41215d47ded1d11967de9d9843be0dbb4333fbf68bf9faf712b13680eed18\": container with ID starting with aeb41215d47ded1d11967de9d9843be0dbb4333fbf68bf9faf712b13680eed18 not found: ID does not exist" containerID="aeb41215d47ded1d11967de9d9843be0dbb4333fbf68bf9faf712b13680eed18"
Nov 03 20:11:21.403294 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:21.403204    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeb41215d47ded1d11967de9d9843be0dbb4333fbf68bf9faf712b13680eed18"} err="failed to get container status \"aeb41215d47ded1d11967de9d9843be0dbb4333fbf68bf9faf712b13680eed18\": rpc error: code = NotFound desc = could not find container \"aeb41215d47ded1d11967de9d9843be0dbb4333fbf68bf9faf712b13680eed18\": container with ID starting with aeb41215d47ded1d11967de9d9843be0dbb4333fbf68bf9faf712b13680eed18 not found: ID does not exist"
Nov 03 20:11:21.415527 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:21.415505    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f8b96f5f6-jjp75"]
Nov 03 20:11:21.419763 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:21.419746    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7f8b96f5f6-jjp75"]
Nov 03 20:11:22.036635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:22.033949    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d1293d5-41dc-4c38-95c9-4af9b577705c" path="/var/lib/kubelet/pods/2d1293d5-41dc-4c38-95c9-4af9b577705c/volumes"
Nov 03 20:11:28.347322 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.347296    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"]
Nov 03 20:11:28.347781 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.347650    4421 cpu_manager.go:414] "containerMap: removing container" podUID="2d1293d5-41dc-4c38-95c9-4af9b577705c" containerName="authorino"
Nov 03 20:11:28.347781 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.347661    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d1293d5-41dc-4c38-95c9-4af9b577705c" containerName="authorino"
Nov 03 20:11:28.347781 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.347741    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d1293d5-41dc-4c38-95c9-4af9b577705c" containerName="authorino"
Nov 03 20:11:28.352417 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.352398    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.365929 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.365906    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"]
Nov 03 20:11:28.366282 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.366257    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-225mv\""
Nov 03 20:11:28.367785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.367764    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"]
Nov 03 20:11:28.371335 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.371307    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.391710 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.391656    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"]
Nov 03 20:11:28.421736 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.421693    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfw4b\" (UniqueName: \"kubernetes.io/projected/9d6e2ee7-7d48-4879-b6fa-f303b5d24697-kube-api-access-mfw4b\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6bhmr\" (UID: \"9d6e2ee7-7d48-4879-b6fa-f303b5d24697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.421908 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.421746    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cd5ll\" (UID: \"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.421908 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.421812    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5mf8\" (UniqueName: \"kubernetes.io/projected/b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d-kube-api-access-n5mf8\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cd5ll\" (UID: \"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.421908 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.421840    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6e2ee7-7d48-4879-b6fa-f303b5d24697-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6bhmr\" (UID: \"9d6e2ee7-7d48-4879-b6fa-f303b5d24697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.421908 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.421856    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d6e2ee7-7d48-4879-b6fa-f303b5d24697-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6bhmr\" (UID: \"9d6e2ee7-7d48-4879-b6fa-f303b5d24697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.422130 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.421957    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6e2ee7-7d48-4879-b6fa-f303b5d24697-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6bhmr\" (UID: \"9d6e2ee7-7d48-4879-b6fa-f303b5d24697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.422130 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.422031    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cd5ll\" (UID: \"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.422130 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.422064    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cd5ll\" (UID: \"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.422130 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.422097    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d6e2ee7-7d48-4879-b6fa-f303b5d24697-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6bhmr\" (UID: \"9d6e2ee7-7d48-4879-b6fa-f303b5d24697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.422130 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.422122    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cd5ll\" (UID: \"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.422306 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.422204    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6e2ee7-7d48-4879-b6fa-f303b5d24697-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6bhmr\" (UID: \"9d6e2ee7-7d48-4879-b6fa-f303b5d24697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.422306 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.422242    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d6e2ee7-7d48-4879-b6fa-f303b5d24697-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6bhmr\" (UID: \"9d6e2ee7-7d48-4879-b6fa-f303b5d24697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.422306 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.422261    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cd5ll\" (UID: \"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.422306 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.422282    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cd5ll\" (UID: \"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.422457 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.422346    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d6e2ee7-7d48-4879-b6fa-f303b5d24697-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6bhmr\" (UID: \"9d6e2ee7-7d48-4879-b6fa-f303b5d24697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.422457 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.422362    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cd5ll\" (UID: \"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.422457 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.422423    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cd5ll\" (UID: \"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.422457 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.422441    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d6e2ee7-7d48-4879-b6fa-f303b5d24697-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6bhmr\" (UID: \"9d6e2ee7-7d48-4879-b6fa-f303b5d24697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.523345 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.523309    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d6e2ee7-7d48-4879-b6fa-f303b5d24697-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6bhmr\" (UID: \"9d6e2ee7-7d48-4879-b6fa-f303b5d24697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.523345 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.523348    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cd5ll\" (UID: \"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.523629 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.523363    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cd5ll\" (UID: \"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.523629 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.523385    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d6e2ee7-7d48-4879-b6fa-f303b5d24697-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6bhmr\" (UID: \"9d6e2ee7-7d48-4879-b6fa-f303b5d24697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.523629 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.523404    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cd5ll\" (UID: \"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.523629 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.523427    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cd5ll\" (UID: \"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.523629 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.523550    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d6e2ee7-7d48-4879-b6fa-f303b5d24697-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6bhmr\" (UID: \"9d6e2ee7-7d48-4879-b6fa-f303b5d24697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.523896 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.523635    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfw4b\" (UniqueName: \"kubernetes.io/projected/9d6e2ee7-7d48-4879-b6fa-f303b5d24697-kube-api-access-mfw4b\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6bhmr\" (UID: \"9d6e2ee7-7d48-4879-b6fa-f303b5d24697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.523896 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.523680    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cd5ll\" (UID: \"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.523896 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.523709    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5mf8\" (UniqueName: \"kubernetes.io/projected/b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d-kube-api-access-n5mf8\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cd5ll\" (UID: \"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.523896 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.523737    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6e2ee7-7d48-4879-b6fa-f303b5d24697-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6bhmr\" (UID: \"9d6e2ee7-7d48-4879-b6fa-f303b5d24697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.523896 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.523760    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d6e2ee7-7d48-4879-b6fa-f303b5d24697-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6bhmr\" (UID: \"9d6e2ee7-7d48-4879-b6fa-f303b5d24697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.523896 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.523788    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6e2ee7-7d48-4879-b6fa-f303b5d24697-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6bhmr\" (UID: \"9d6e2ee7-7d48-4879-b6fa-f303b5d24697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.523896 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.523824    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cd5ll\" (UID: \"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.523896 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.523853    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cd5ll\" (UID: \"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.523896 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.523898    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d6e2ee7-7d48-4879-b6fa-f303b5d24697-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6bhmr\" (UID: \"9d6e2ee7-7d48-4879-b6fa-f303b5d24697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.524349 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.523904    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d6e2ee7-7d48-4879-b6fa-f303b5d24697-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6bhmr\" (UID: \"9d6e2ee7-7d48-4879-b6fa-f303b5d24697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.524349 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.523922    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cd5ll\" (UID: \"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.524349 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.523982    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6e2ee7-7d48-4879-b6fa-f303b5d24697-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6bhmr\" (UID: \"9d6e2ee7-7d48-4879-b6fa-f303b5d24697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.524349 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.524145    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6e2ee7-7d48-4879-b6fa-f303b5d24697-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6bhmr\" (UID: \"9d6e2ee7-7d48-4879-b6fa-f303b5d24697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.524349 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.524210    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d6e2ee7-7d48-4879-b6fa-f303b5d24697-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6bhmr\" (UID: \"9d6e2ee7-7d48-4879-b6fa-f303b5d24697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.524349 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.524320    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d6e2ee7-7d48-4879-b6fa-f303b5d24697-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6bhmr\" (UID: \"9d6e2ee7-7d48-4879-b6fa-f303b5d24697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.524349 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.523909    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cd5ll\" (UID: \"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.524745 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.524349    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cd5ll\" (UID: \"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.524745 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.524543    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d6e2ee7-7d48-4879-b6fa-f303b5d24697-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6bhmr\" (UID: \"9d6e2ee7-7d48-4879-b6fa-f303b5d24697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.524853 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.524762    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cd5ll\" (UID: \"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.524994 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.524971    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cd5ll\" (UID: \"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.525066 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.525052    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cd5ll\" (UID: \"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.526050 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.526025    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cd5ll\" (UID: \"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.526562 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.526541    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d6e2ee7-7d48-4879-b6fa-f303b5d24697-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6bhmr\" (UID: \"9d6e2ee7-7d48-4879-b6fa-f303b5d24697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.526757 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.526739    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cd5ll\" (UID: \"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.526957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.526936    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d6e2ee7-7d48-4879-b6fa-f303b5d24697-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6bhmr\" (UID: \"9d6e2ee7-7d48-4879-b6fa-f303b5d24697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.536611 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.536568    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d6e2ee7-7d48-4879-b6fa-f303b5d24697-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6bhmr\" (UID: \"9d6e2ee7-7d48-4879-b6fa-f303b5d24697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.536784 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.536762    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cd5ll\" (UID: \"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.537046 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.537026    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfw4b\" (UniqueName: \"kubernetes.io/projected/9d6e2ee7-7d48-4879-b6fa-f303b5d24697-kube-api-access-mfw4b\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6bhmr\" (UID: \"9d6e2ee7-7d48-4879-b6fa-f303b5d24697\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.537106 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.537041    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5mf8\" (UniqueName: \"kubernetes.io/projected/b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d-kube-api-access-n5mf8\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cd5ll\" (UID: \"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.666752 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.666685    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:28.684904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.684880    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:28.947410 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.947387    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"]
Nov 03 20:11:28.949588 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:11:28.949558    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d6e2ee7_7d48_4879_b6fa_f303b5d24697.slice/crio-fd798abfc6768d13481e3adb97681b2245f95d7e1c6baf84850c6db627ffe33d WatchSource:0}: Error finding container fd798abfc6768d13481e3adb97681b2245f95d7e1c6baf84850c6db627ffe33d: Status 404 returned error can't find the container with id fd798abfc6768d13481e3adb97681b2245f95d7e1c6baf84850c6db627ffe33d
Nov 03 20:11:28.970050 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:28.970028    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"]
Nov 03 20:11:28.971796 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:11:28.971773    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a7c2ad_1827_4aa9_a327_aa1ac8bc2b0d.slice/crio-a0ffa8d4ec1248931295b57875ff4c87749103ebbdb66ca8588a87ef488d60a6 WatchSource:0}: Error finding container a0ffa8d4ec1248931295b57875ff4c87749103ebbdb66ca8588a87ef488d60a6: Status 404 returned error can't find the container with id a0ffa8d4ec1248931295b57875ff4c87749103ebbdb66ca8588a87ef488d60a6
Nov 03 20:11:29.433251 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:29.433159    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll" event={"ID":"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d","Type":"ContainerStarted","Data":"a0ffa8d4ec1248931295b57875ff4c87749103ebbdb66ca8588a87ef488d60a6"}
Nov 03 20:11:29.434377 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:29.434353    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr" event={"ID":"9d6e2ee7-7d48-4879-b6fa-f303b5d24697","Type":"ContainerStarted","Data":"fd798abfc6768d13481e3adb97681b2245f95d7e1c6baf84850c6db627ffe33d"}
Nov 03 20:11:32.615256 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:32.615225    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:11:32.615574 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:32.615291    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:11:32.615574 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:32.615327    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:11:33.038075 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:33.038038    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:11:33.038145 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:33.038117    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:11:33.038185 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:33.038150    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:11:33.474016 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:33.473975    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr" event={"ID":"9d6e2ee7-7d48-4879-b6fa-f303b5d24697","Type":"ContainerStarted","Data":"187e034fee3eb9956dbe04327033cc6aa3685ec5d19f133c79bc4c4a09cdd334"}
Nov 03 20:11:33.475659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:33.475582    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll" event={"ID":"b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d","Type":"ContainerStarted","Data":"8104f9fb65c3e02105801f4511571d9221bf9f20f25111d5f86656ec7b80ec2c"}
Nov 03 20:11:33.498731 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:33.498671    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr" podStartSLOduration=1.835938582 podStartE2EDuration="5.498648647s" totalImagesPullingTime="3.662710065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:33.49734887 +0000 UTC m=+623.630477332" watchObservedRunningTime="2026-11-03 20:11:33.498648647 +0000 UTC m=+623.631777111"
Nov 03 20:11:33.521771 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:33.521713    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll" podStartSLOduration=1.458149616 podStartE2EDuration="5.52169425s" totalImagesPullingTime="4.063544634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:33.520445879 +0000 UTC m=+623.653574342" watchObservedRunningTime="2026-11-03 20:11:33.52169425 +0000 UTC m=+623.654822711"
Nov 03 20:11:33.668626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:33.667957    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:33.675983 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:33.675941    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-6bhmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 20:11:33.676294 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:33.676262    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr" podUID="9d6e2ee7-7d48-4879-b6fa-f303b5d24697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 20:11:33.687124 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:33.687092    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:34.675780 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:34.675746    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-6bhmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 20:11:34.676410 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:34.676358    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr" podUID="9d6e2ee7-7d48-4879-b6fa-f303b5d24697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 20:11:34.692639 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:34.691711    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-cd5ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 20:11:34.692639 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:34.691759    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll" podUID="b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 20:11:35.671623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:35.670983    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-6bhmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 20:11:35.671623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:35.671045    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr" podUID="9d6e2ee7-7d48-4879-b6fa-f303b5d24697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 20:11:35.691095 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:35.691053    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-cd5ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 20:11:35.691561 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:35.691533    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll" podUID="b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 20:11:36.674047 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:36.674008    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-6bhmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 20:11:36.674345 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:36.674313    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr" podUID="9d6e2ee7-7d48-4879-b6fa-f303b5d24697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 20:11:36.685437 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:36.685409    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-cd5ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 20:11:36.685674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:36.685646    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll" podUID="b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 20:11:37.671623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:37.669962    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-6bhmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 20:11:37.671623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:37.670024    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr" podUID="9d6e2ee7-7d48-4879-b6fa-f303b5d24697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 20:11:37.686057 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:37.686024    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-cd5ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 20:11:37.686322 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:37.686286    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll" podUID="b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 20:11:38.668256 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:38.667822    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-6bhmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 20:11:38.668256 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:38.667881    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr" podUID="9d6e2ee7-7d48-4879-b6fa-f303b5d24697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 20:11:38.686052 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:38.686017    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-cd5ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 20:11:38.686526 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:38.686064    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll" podUID="b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 20:11:39.667200 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:39.667159    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-6bhmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 20:11:39.667407 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:39.667229    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr" podUID="9d6e2ee7-7d48-4879-b6fa-f303b5d24697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 20:11:39.685283 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:39.685245    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-cd5ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 20:11:39.685438 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:39.685302    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll" podUID="b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 20:11:40.686209 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:40.686171    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-cd5ll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 20:11:40.686619 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:40.686239    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll" podUID="b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 20:11:41.443354 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:41.442611    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:41.443354 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:41.442962    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:41.443785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:41.443767    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr"
Nov 03 20:11:41.880782 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:41.880692    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:41.881195 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:41.881022    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:41.881782 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:41.881757    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll"
Nov 03 20:11:43.381946 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.381905    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"]
Nov 03 20:11:43.386125 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.386102    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.405052 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.405026    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"]
Nov 03 20:11:43.479191 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.479154    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-w9chd\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.479356 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.479213    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxh57\" (UniqueName: \"kubernetes.io/projected/657622a4-6298-4496-b2c3-edce95190d04-kube-api-access-dxh57\") pod \"maas-default-gateway-openshift-default-59f957fdd7-w9chd\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.479356 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.479324    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-w9chd\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.479482 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.479361    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/657622a4-6298-4496-b2c3-edce95190d04-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-w9chd\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.479482 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.479388    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-w9chd\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.479482 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.479413    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-w9chd\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.479482 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.479448    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657622a4-6298-4496-b2c3-edce95190d04-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-w9chd\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.479690 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.479532    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657622a4-6298-4496-b2c3-edce95190d04-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-w9chd\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.479690 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.479570    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-w9chd\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.580763 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.580736    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-w9chd\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.580763 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.580768    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/657622a4-6298-4496-b2c3-edce95190d04-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-w9chd\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.580999 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.580784    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-w9chd\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.580999 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.580803    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-w9chd\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.580999 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.580828    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657622a4-6298-4496-b2c3-edce95190d04-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-w9chd\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.580999 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.580982    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657622a4-6298-4496-b2c3-edce95190d04-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-w9chd\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.581201 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.581044    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-w9chd\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.581201 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.581127    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-w9chd\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.581302 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.581209    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxh57\" (UniqueName: \"kubernetes.io/projected/657622a4-6298-4496-b2c3-edce95190d04-kube-api-access-dxh57\") pod \"maas-default-gateway-openshift-default-59f957fdd7-w9chd\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.581302 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.581231    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-w9chd\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.581302 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.581269    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-w9chd\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.581547 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.581525    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-w9chd\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.581742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.581524    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-w9chd\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.581742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.581655    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657622a4-6298-4496-b2c3-edce95190d04-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-w9chd\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.583587 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.583561    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657622a4-6298-4496-b2c3-edce95190d04-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-w9chd\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.583710 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.583563    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-w9chd\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.593403 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.593383    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/657622a4-6298-4496-b2c3-edce95190d04-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-w9chd\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.593545 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.593528    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxh57\" (UniqueName: \"kubernetes.io/projected/657622a4-6298-4496-b2c3-edce95190d04-kube-api-access-dxh57\") pod \"maas-default-gateway-openshift-default-59f957fdd7-w9chd\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.704155 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.704129    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:43.984059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.984032    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"]
Nov 03 20:11:43.985307 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:11:43.985279    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657622a4_6298_4496_b2c3_edce95190d04.slice/crio-d890e08fb82c5b58c3e22915fd3e75da349ca675c72a133458f422eb9ee4e558 WatchSource:0}: Error finding container d890e08fb82c5b58c3e22915fd3e75da349ca675c72a133458f422eb9ee4e558: Status 404 returned error can't find the container with id d890e08fb82c5b58c3e22915fd3e75da349ca675c72a133458f422eb9ee4e558
Nov 03 20:11:43.987510 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.987477    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:11:43.987622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.987539    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:11:43.987622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:43.987570    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:11:44.546097 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:44.546055    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd" event={"ID":"657622a4-6298-4496-b2c3-edce95190d04","Type":"ContainerStarted","Data":"b782e8874dc9f21a856019f11c2650c2ae2f54b007300ca1d318df51d801f9d2"}
Nov 03 20:11:44.546731 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:44.546700    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd" event={"ID":"657622a4-6298-4496-b2c3-edce95190d04","Type":"ContainerStarted","Data":"d890e08fb82c5b58c3e22915fd3e75da349ca675c72a133458f422eb9ee4e558"}
Nov 03 20:11:44.580555 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:44.580503    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd" podStartSLOduration=1.580489523 podStartE2EDuration="1.580489523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:44.578538786 +0000 UTC m=+634.711667248" watchObservedRunningTime="2026-11-03 20:11:44.580489523 +0000 UTC m=+634.713618054"
Nov 03 20:11:44.704754 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:44.704719    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:45.705417 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:45.705375    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-w9chd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 20:11:45.705838 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:45.705428    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd" podUID="657622a4-6298-4496-b2c3-edce95190d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 20:11:46.704816 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:46.704778    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-w9chd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 20:11:46.705010 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:46.704832    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd" podUID="657622a4-6298-4496-b2c3-edce95190d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 20:11:47.705421 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:47.705390    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-w9chd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 20:11:47.705801 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:47.705442    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd" podUID="657622a4-6298-4496-b2c3-edce95190d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 20:11:48.704901 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:48.704864    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-w9chd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 20:11:48.705079 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:48.704916    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd" podUID="657622a4-6298-4496-b2c3-edce95190d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 20:11:49.628217 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:11:49.628175    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:11:49.705373 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:49.705345    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-w9chd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 20:11:49.705531 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:49.705393    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd" podUID="657622a4-6298-4496-b2c3-edce95190d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 20:11:50.133761 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.133728    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"]
Nov 03 20:11:50.138560 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.138531    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.142727 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.142702    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9dd42e13-openshift-default-dockercfg-mkrv9\""
Nov 03 20:11:50.150366 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.150342    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"]
Nov 03 20:11:50.245367 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.245331    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-workload-socket\") pod \"e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") " pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.245669 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.245637    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-workload-certs\") pod \"e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") " pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.245829 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.245707    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-data\") pod \"e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") " pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.245829 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.245801    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-token\") pod \"e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") " pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.245955 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.245868    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t5w7\" (UniqueName: \"kubernetes.io/projected/ecbd165b-b0c7-436d-b1ef-6dae96349766-kube-api-access-5t5w7\") pod \"e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") " pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.245955 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.245902    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-credential-socket\") pod \"e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") " pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.245955 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.245928    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecbd165b-b0c7-436d-b1ef-6dae96349766-istiod-ca-cert\") pod \"e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") " pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.246121 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.245983    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-envoy\") pod \"e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") " pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.246121 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.246062    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-podinfo\") pod \"e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") " pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.354647 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.350899    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-envoy\") pod \"e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") " pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.354647 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.350984    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-podinfo\") pod \"e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") " pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.354647 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.351034    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-workload-socket\") pod \"e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") " pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.354647 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.351069    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-workload-certs\") pod \"e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") " pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.354647 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.351096    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-data\") pod \"e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") " pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.354647 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.351116    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-token\") pod \"e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") " pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.354647 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.351145    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t5w7\" (UniqueName: \"kubernetes.io/projected/ecbd165b-b0c7-436d-b1ef-6dae96349766-kube-api-access-5t5w7\") pod \"e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") " pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.354647 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.351163    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-credential-socket\") pod \"e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") " pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.354647 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.351177    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecbd165b-b0c7-436d-b1ef-6dae96349766-istiod-ca-cert\") pod \"e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") " pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.354647 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.351954    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-workload-certs\") pod \"e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") " pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.354647 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.352196    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-workload-socket\") pod \"e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") " pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.354647 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.352409    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-data\") pod \"e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") " pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.354647 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.352712    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-credential-socket\") pod \"e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") " pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.354647 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.352988    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecbd165b-b0c7-436d-b1ef-6dae96349766-istiod-ca-cert\") pod \"e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") " pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.356570 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.356283    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-podinfo\") pod \"e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") " pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.369549 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.369523    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-token\") pod \"e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") " pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.371652 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.371627    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-envoy\") pod \"e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") " pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.384804 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.384735    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t5w7\" (UniqueName: \"kubernetes.io/projected/ecbd165b-b0c7-436d-b1ef-6dae96349766-kube-api-access-5t5w7\") pod \"e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") " pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.460567 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.460524    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:50.560746 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.560709    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"]
Nov 03 20:11:50.565769 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.565746    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.580200 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.580169    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"]
Nov 03 20:11:50.645219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.645142    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-459ca04a-gw-openshift-default-dockercfg-26665\""
Nov 03 20:11:50.654376 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.654349    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-workload-certs\") pod \"e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") " pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.654504 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.654401    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/172103a4-4b03-4580-a3a9-2cd8a744b147-istiod-ca-cert\") pod \"e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") " pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.654504 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.654457    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42rrt\" (UniqueName: \"kubernetes.io/projected/172103a4-4b03-4580-a3a9-2cd8a744b147-kube-api-access-42rrt\") pod \"e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") " pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.654574 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.654530    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-workload-socket\") pod \"e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") " pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.654628 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.654579    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-podinfo\") pod \"e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") " pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.654660 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.654648    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-envoy\") pod \"e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") " pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.654698 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.654686    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-data\") pod \"e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") " pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.654743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.654730    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-credential-socket\") pod \"e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") " pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.654781 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.654755    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-token\") pod \"e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") " pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.704757 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.704722    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-w9chd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 20:11:50.704883 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.704777    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd" podUID="657622a4-6298-4496-b2c3-edce95190d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 20:11:50.746574 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.746542    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"]
Nov 03 20:11:50.747530 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:11:50.747500    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecbd165b_b0c7_436d_b1ef_6dae96349766.slice/crio-545dfa3ccf251be19e08618c07ee7d17017d2010e7fcef6cbcc87c95de2883f9 WatchSource:0}: Error finding container 545dfa3ccf251be19e08618c07ee7d17017d2010e7fcef6cbcc87c95de2883f9: Status 404 returned error can't find the container with id 545dfa3ccf251be19e08618c07ee7d17017d2010e7fcef6cbcc87c95de2883f9
Nov 03 20:11:50.762512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.750736    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:11:50.762512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.750831    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:11:50.762512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.750895    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:11:50.763308 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.763274    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-envoy\") pod \"e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") " pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.763439 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.763336    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-data\") pod \"e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") " pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.763439 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.763374    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-credential-socket\") pod \"e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") " pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.763439 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.763402    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-token\") pod \"e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") " pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.763642 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.763475    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-workload-certs\") pod \"e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") " pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.763642 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.763515    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/172103a4-4b03-4580-a3a9-2cd8a744b147-istiod-ca-cert\") pod \"e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") " pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.763642 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.763551    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42rrt\" (UniqueName: \"kubernetes.io/projected/172103a4-4b03-4580-a3a9-2cd8a744b147-kube-api-access-42rrt\") pod \"e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") " pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.763642 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.763584    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-workload-socket\") pod \"e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") " pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.763841 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.763655    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-podinfo\") pod \"e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") " pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.770636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.766681    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-podinfo\") pod \"e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") " pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.770636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.766863    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-workload-certs\") pod \"e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") " pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.770636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.767264    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/172103a4-4b03-4580-a3a9-2cd8a744b147-istiod-ca-cert\") pod \"e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") " pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.770636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.767653    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-workload-socket\") pod \"e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") " pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.770636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.767752    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-envoy\") pod \"e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") " pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.770636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.767900    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-data\") pod \"e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") " pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.770636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.768105    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-credential-socket\") pod \"e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") " pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.778096 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.778078    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-token\") pod \"e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") " pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.781337 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.781312    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42rrt\" (UniqueName: \"kubernetes.io/projected/172103a4-4b03-4580-a3a9-2cd8a744b147-kube-api-access-42rrt\") pod \"e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") " pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:50.884678 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:50.884642    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:51.192124 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:51.192094    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"]
Nov 03 20:11:51.194004 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:11:51.193979    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod172103a4_4b03_4580_a3a9_2cd8a744b147.slice/crio-1ec08a1aa91ad15d17e5786d7d358adec4949292e65a33446edaf14548caa595 WatchSource:0}: Error finding container 1ec08a1aa91ad15d17e5786d7d358adec4949292e65a33446edaf14548caa595: Status 404 returned error can't find the container with id 1ec08a1aa91ad15d17e5786d7d358adec4949292e65a33446edaf14548caa595
Nov 03 20:11:51.196730 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:51.196699    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:11:51.196815 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:51.196758    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:11:51.196815 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:51.196789    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:11:51.604667 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:51.604567    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm" event={"ID":"ecbd165b-b0c7-436d-b1ef-6dae96349766","Type":"ContainerStarted","Data":"fceba77512b24390bbb55cedf987f2684ea797234fc9972a11576f2e046fb96d"}
Nov 03 20:11:51.604667 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:51.604652    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm" event={"ID":"ecbd165b-b0c7-436d-b1ef-6dae96349766","Type":"ContainerStarted","Data":"545dfa3ccf251be19e08618c07ee7d17017d2010e7fcef6cbcc87c95de2883f9"}
Nov 03 20:11:51.606218 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:51.606189    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm" event={"ID":"172103a4-4b03-4580-a3a9-2cd8a744b147","Type":"ContainerStarted","Data":"043ae63f76d2328d9fa44e016c445232f1d8258c034779c99acce7ec8a8e676f"}
Nov 03 20:11:51.606342 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:51.606241    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm" event={"ID":"172103a4-4b03-4580-a3a9-2cd8a744b147","Type":"ContainerStarted","Data":"1ec08a1aa91ad15d17e5786d7d358adec4949292e65a33446edaf14548caa595"}
Nov 03 20:11:51.628609 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:51.628562    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm" podStartSLOduration=1.628549811 podStartE2EDuration="1.628549811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:51.626877415 +0000 UTC m=+641.760005877" watchObservedRunningTime="2026-11-03 20:11:51.628549811 +0000 UTC m=+641.761678273"
Nov 03 20:11:51.673108 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:51.673055    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm" podStartSLOduration=1.673040404 podStartE2EDuration="1.673040404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:51.670188751 +0000 UTC m=+641.803317215" watchObservedRunningTime="2026-11-03 20:11:51.673040404 +0000 UTC m=+641.806168865"
Nov 03 20:11:51.705212 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:51.705186    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-w9chd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 20:11:51.705338 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:51.705233    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd" podUID="657622a4-6298-4496-b2c3-edce95190d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 20:11:51.885303 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:51.885219    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:52.462537 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:52.461734    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:52.467488 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:52.467460    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:52.612437 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:52.612402    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:52.613549 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:52.613524    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:11:52.890817 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:52.890744    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:52.915722 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:52.914653    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:52.915722 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:52.915457    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:52.916581 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:52.916556    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 20:11:53.616961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:53.616926    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:53.617900 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:53.617881    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:11:53.845496 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:53.845463    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d"]
Nov 03 20:11:53.851039 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:53.851009    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d"
Nov 03 20:11:53.856674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:53.856650    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-459ca04a-serving-cert\""
Nov 03 20:11:53.859151 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:53.859129    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d"]
Nov 03 20:11:54.025918 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:54.025886    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-metrics-certs\") pod \"maas-api-e2e-migrate-459ca04a-5885b88747-ld96d\" (UID: \"a6ae90b7-a7bf-4dc3-b132-020d7a4044e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d"
Nov 03 20:11:54.026340 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:54.025930    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sthfc\" (UniqueName: \"kubernetes.io/projected/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-kube-api-access-sthfc\") pod \"maas-api-e2e-migrate-459ca04a-5885b88747-ld96d\" (UID: \"a6ae90b7-a7bf-4dc3-b132-020d7a4044e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d"
Nov 03 20:11:54.026340 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:54.026032    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-maas-api-tls\") pod \"maas-api-e2e-migrate-459ca04a-5885b88747-ld96d\" (UID: \"a6ae90b7-a7bf-4dc3-b132-020d7a4044e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d"
Nov 03 20:11:54.026340 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:54.026167    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-openshift-service-ca\") pod \"maas-api-e2e-migrate-459ca04a-5885b88747-ld96d\" (UID: \"a6ae90b7-a7bf-4dc3-b132-020d7a4044e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d"
Nov 03 20:11:54.127403 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:54.127369    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-metrics-certs\") pod \"maas-api-e2e-migrate-459ca04a-5885b88747-ld96d\" (UID: \"a6ae90b7-a7bf-4dc3-b132-020d7a4044e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d"
Nov 03 20:11:54.127571 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:54.127424    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sthfc\" (UniqueName: \"kubernetes.io/projected/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-kube-api-access-sthfc\") pod \"maas-api-e2e-migrate-459ca04a-5885b88747-ld96d\" (UID: \"a6ae90b7-a7bf-4dc3-b132-020d7a4044e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d"
Nov 03 20:11:54.127571 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:54.127472    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-maas-api-tls\") pod \"maas-api-e2e-migrate-459ca04a-5885b88747-ld96d\" (UID: \"a6ae90b7-a7bf-4dc3-b132-020d7a4044e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d"
Nov 03 20:11:54.127571 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:54.127539    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-openshift-service-ca\") pod \"maas-api-e2e-migrate-459ca04a-5885b88747-ld96d\" (UID: \"a6ae90b7-a7bf-4dc3-b132-020d7a4044e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d"
Nov 03 20:11:54.128193 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:54.128168    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-openshift-service-ca\") pod \"maas-api-e2e-migrate-459ca04a-5885b88747-ld96d\" (UID: \"a6ae90b7-a7bf-4dc3-b132-020d7a4044e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d"
Nov 03 20:11:54.130025 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:54.130006    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-metrics-certs\") pod \"maas-api-e2e-migrate-459ca04a-5885b88747-ld96d\" (UID: \"a6ae90b7-a7bf-4dc3-b132-020d7a4044e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d"
Nov 03 20:11:54.130101 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:54.130052    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-maas-api-tls\") pod \"maas-api-e2e-migrate-459ca04a-5885b88747-ld96d\" (UID: \"a6ae90b7-a7bf-4dc3-b132-020d7a4044e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d"
Nov 03 20:11:54.142305 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:54.142277    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sthfc\" (UniqueName: \"kubernetes.io/projected/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-kube-api-access-sthfc\") pod \"maas-api-e2e-migrate-459ca04a-5885b88747-ld96d\" (UID: \"a6ae90b7-a7bf-4dc3-b132-020d7a4044e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d"
Nov 03 20:11:54.175733 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:54.175708    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d"
Nov 03 20:11:54.466336 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:54.466309    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d"]
Nov 03 20:11:54.467532 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:11:54.467507    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ae90b7_a7bf_4dc3_b132_020d7a4044e8.slice/crio-97581bf4491b4e28a60bf0b5adc7ab4e8f071f4018ae56c4035580e968df8df9 WatchSource:0}: Error finding container 97581bf4491b4e28a60bf0b5adc7ab4e8f071f4018ae56c4035580e968df8df9: Status 404 returned error can't find the container with id 97581bf4491b4e28a60bf0b5adc7ab4e8f071f4018ae56c4035580e968df8df9
Nov 03 20:11:54.468929 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:54.468915    4421 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:11:54.624146 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:54.624113    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d" event={"ID":"a6ae90b7-a7bf-4dc3-b132-020d7a4044e8","Type":"ContainerStarted","Data":"97581bf4491b4e28a60bf0b5adc7ab4e8f071f4018ae56c4035580e968df8df9"}
Nov 03 20:11:55.057155 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:55.057130    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw"]
Nov 03 20:11:55.060681 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:55.060661    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw"
Nov 03 20:11:55.083855 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:55.083830    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e17fc195\""
Nov 03 20:11:55.086121 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:55.085968    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw"]
Nov 03 20:11:55.145483 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:55.145441    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981a2ed3-5af9-45ff-b165-283348d9942e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw\" (UID: \"981a2ed3-5af9-45ff-b165-283348d9942e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw"
Nov 03 20:11:55.145683 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:55.145616    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fbfg\" (UniqueName: \"kubernetes.io/projected/981a2ed3-5af9-45ff-b165-283348d9942e-kube-api-access-9fbfg\") pod \"payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw\" (UID: \"981a2ed3-5af9-45ff-b165-283348d9942e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw"
Nov 03 20:11:55.160876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:55.160844    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e17fc195-dockercfg-gvbw4\""
Nov 03 20:11:55.250668 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:55.250143    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fbfg\" (UniqueName: \"kubernetes.io/projected/981a2ed3-5af9-45ff-b165-283348d9942e-kube-api-access-9fbfg\") pod \"payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw\" (UID: \"981a2ed3-5af9-45ff-b165-283348d9942e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw"
Nov 03 20:11:55.250668 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:55.250303    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981a2ed3-5af9-45ff-b165-283348d9942e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw\" (UID: \"981a2ed3-5af9-45ff-b165-283348d9942e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw"
Nov 03 20:11:55.254633 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:55.251246    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981a2ed3-5af9-45ff-b165-283348d9942e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw\" (UID: \"981a2ed3-5af9-45ff-b165-283348d9942e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw"
Nov 03 20:11:55.264765 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:55.264693    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fbfg\" (UniqueName: \"kubernetes.io/projected/981a2ed3-5af9-45ff-b165-283348d9942e-kube-api-access-9fbfg\") pod \"payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw\" (UID: \"981a2ed3-5af9-45ff-b165-283348d9942e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw"
Nov 03 20:11:55.374266 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:55.374209    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw"
Nov 03 20:11:55.629751 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:55.629649    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d" event={"ID":"a6ae90b7-a7bf-4dc3-b132-020d7a4044e8","Type":"ContainerStarted","Data":"8e74a7e5a0513c714729218e260e0ac16a4e0cd1ecff7eda675fa1fe6aea8db5"}
Nov 03 20:11:55.629908 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:55.629769    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d"
Nov 03 20:11:55.660021 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:55.659990    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw"]
Nov 03 20:11:55.662537 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:55.662497    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d" podStartSLOduration=2.132638045 podStartE2EDuration="2.662484767s" totalImagesPullingTime="529.846722ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:55.660431334 +0000 UTC m=+645.793559797" watchObservedRunningTime="2026-11-03 20:11:55.662484767 +0000 UTC m=+645.795613231"
Nov 03 20:11:55.664074 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:11:55.664051    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod981a2ed3_5af9_45ff_b165_283348d9942e.slice/crio-75029bf37fde3c0e6c37f39bbd0e8ad850e26805d5dea9f6938e16d48f01b859 WatchSource:0}: Error finding container 75029bf37fde3c0e6c37f39bbd0e8ad850e26805d5dea9f6938e16d48f01b859: Status 404 returned error can't find the container with id 75029bf37fde3c0e6c37f39bbd0e8ad850e26805d5dea9f6938e16d48f01b859
Nov 03 20:11:55.745455 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:55.745428    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth"]
Nov 03 20:11:55.750175 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:55.750152    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth"
Nov 03 20:11:55.759636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:55.759210    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth"]
Nov 03 20:11:55.810954 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:55.810912    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9dd42e13-dockercfg-r9zmp\""
Nov 03 20:11:55.814160 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:55.813915    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9dd42e13\""
Nov 03 20:11:55.858721 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:55.858675    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19980a3-d265-46e3-be8d-88968dd205fc-plugins-config-volume\") pod \"payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth\" (UID: \"d19980a3-d265-46e3-be8d-88968dd205fc\") " pod="openshift-ingress/payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth"
Nov 03 20:11:55.858893 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:55.858795    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk2t7\" (UniqueName: \"kubernetes.io/projected/d19980a3-d265-46e3-be8d-88968dd205fc-kube-api-access-pk2t7\") pod \"payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth\" (UID: \"d19980a3-d265-46e3-be8d-88968dd205fc\") " pod="openshift-ingress/payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth"
Nov 03 20:11:55.959535 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:55.959495    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19980a3-d265-46e3-be8d-88968dd205fc-plugins-config-volume\") pod \"payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth\" (UID: \"d19980a3-d265-46e3-be8d-88968dd205fc\") " pod="openshift-ingress/payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth"
Nov 03 20:11:55.959750 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:55.959610    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk2t7\" (UniqueName: \"kubernetes.io/projected/d19980a3-d265-46e3-be8d-88968dd205fc-kube-api-access-pk2t7\") pod \"payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth\" (UID: \"d19980a3-d265-46e3-be8d-88968dd205fc\") " pod="openshift-ingress/payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth"
Nov 03 20:11:55.960270 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:55.960244    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19980a3-d265-46e3-be8d-88968dd205fc-plugins-config-volume\") pod \"payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth\" (UID: \"d19980a3-d265-46e3-be8d-88968dd205fc\") " pod="openshift-ingress/payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth"
Nov 03 20:11:55.971821 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:55.971798    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk2t7\" (UniqueName: \"kubernetes.io/projected/d19980a3-d265-46e3-be8d-88968dd205fc-kube-api-access-pk2t7\") pod \"payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth\" (UID: \"d19980a3-d265-46e3-be8d-88968dd205fc\") " pod="openshift-ingress/payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth"
Nov 03 20:11:56.067926 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:56.067889    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth"
Nov 03 20:11:56.371671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:56.360167    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth"]
Nov 03 20:11:56.642326 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:56.641206    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw" event={"ID":"981a2ed3-5af9-45ff-b165-283348d9942e","Type":"ContainerStarted","Data":"357133aaaf35ab2c5fe4092fdea75c5a5bd5ca8370705f1d2b4c1c0d6c3d925f"}
Nov 03 20:11:56.642326 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:56.641317    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw" event={"ID":"981a2ed3-5af9-45ff-b165-283348d9942e","Type":"ContainerStarted","Data":"75029bf37fde3c0e6c37f39bbd0e8ad850e26805d5dea9f6938e16d48f01b859"}
Nov 03 20:11:56.642326 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:56.642228    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw"
Nov 03 20:11:56.646316 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:56.645724    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth" event={"ID":"d19980a3-d265-46e3-be8d-88968dd205fc","Type":"ContainerStarted","Data":"f124aa3695f2fa69196ad21e8814480dbd00b7382b8333dad424f91705ceaec1"}
Nov 03 20:11:56.646316 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:56.645770    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth" event={"ID":"d19980a3-d265-46e3-be8d-88968dd205fc","Type":"ContainerStarted","Data":"a0ad7f6e3803a6873cfc636b31772fba3cce09536713d9e1eaa2182b64116c53"}
Nov 03 20:11:56.673927 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:56.673875    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw" podStartSLOduration=1.6738607650000001 podStartE2EDuration="1.673860765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:56.672429005 +0000 UTC m=+646.805557469" watchObservedRunningTime="2026-11-03 20:11:56.673860765 +0000 UTC m=+646.806989227"
Nov 03 20:11:56.697248 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:56.697193    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth" podStartSLOduration=1.6971805089999998 podStartE2EDuration="1.697180509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:56.69474861 +0000 UTC m=+646.827877073" watchObservedRunningTime="2026-11-03 20:11:56.697180509 +0000 UTC m=+646.830309019"
Nov 03 20:11:56.736675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:56.736643    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm"]
Nov 03 20:11:56.740510 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:56.740491    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm"
Nov 03 20:11:56.749552 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:56.749529    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4bd752fd\""
Nov 03 20:11:56.752629 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:56.751819    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm"]
Nov 03 20:11:56.791447 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:56.791355    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4bd752fd-dockercfg-6f27b\""
Nov 03 20:11:56.895103 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:56.895006    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f88f1ea1-6c54-4741-bb9c-0676f02b6ff1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm\" (UID: \"f88f1ea1-6c54-4741-bb9c-0676f02b6ff1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm"
Nov 03 20:11:56.895103 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:56.895077    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtmq8\" (UniqueName: \"kubernetes.io/projected/f88f1ea1-6c54-4741-bb9c-0676f02b6ff1-kube-api-access-rtmq8\") pod \"payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm\" (UID: \"f88f1ea1-6c54-4741-bb9c-0676f02b6ff1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm"
Nov 03 20:11:56.996821 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:56.996770    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f88f1ea1-6c54-4741-bb9c-0676f02b6ff1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm\" (UID: \"f88f1ea1-6c54-4741-bb9c-0676f02b6ff1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm"
Nov 03 20:11:56.996978 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:56.996830    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtmq8\" (UniqueName: \"kubernetes.io/projected/f88f1ea1-6c54-4741-bb9c-0676f02b6ff1-kube-api-access-rtmq8\") pod \"payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm\" (UID: \"f88f1ea1-6c54-4741-bb9c-0676f02b6ff1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm"
Nov 03 20:11:56.997624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:56.997582    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f88f1ea1-6c54-4741-bb9c-0676f02b6ff1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm\" (UID: \"f88f1ea1-6c54-4741-bb9c-0676f02b6ff1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm"
Nov 03 20:11:57.009673 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:57.009647    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtmq8\" (UniqueName: \"kubernetes.io/projected/f88f1ea1-6c54-4741-bb9c-0676f02b6ff1-kube-api-access-rtmq8\") pod \"payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm\" (UID: \"f88f1ea1-6c54-4741-bb9c-0676f02b6ff1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm"
Nov 03 20:11:57.054801 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:57.054776    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm"
Nov 03 20:11:57.334153 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:57.334123    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm"]
Nov 03 20:11:57.336140 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:11:57.336107    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf88f1ea1_6c54_4741_bb9c_0676f02b6ff1.slice/crio-2e1a62b80486927e1a336f42acc9902b4223ca4d0815d9422f583fbf23e222a5 WatchSource:0}: Error finding container 2e1a62b80486927e1a336f42acc9902b4223ca4d0815d9422f583fbf23e222a5: Status 404 returned error can't find the container with id 2e1a62b80486927e1a336f42acc9902b4223ca4d0815d9422f583fbf23e222a5
Nov 03 20:11:57.673735 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:57.673576    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm" event={"ID":"f88f1ea1-6c54-4741-bb9c-0676f02b6ff1","Type":"ContainerStarted","Data":"baf1581fc7392b77554607551928601006c08a22a3106eb674efb4cbc928de8b"}
Nov 03 20:11:57.673735 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:57.673677    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm"
Nov 03 20:11:57.673735 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:57.673701    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm" event={"ID":"f88f1ea1-6c54-4741-bb9c-0676f02b6ff1","Type":"ContainerStarted","Data":"2e1a62b80486927e1a336f42acc9902b4223ca4d0815d9422f583fbf23e222a5"}
Nov 03 20:11:57.674105 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:57.674086    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth"
Nov 03 20:11:57.693721 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:57.693664    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm" podStartSLOduration=1.6936519799999998 podStartE2EDuration="1.69365198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:57.690968329 +0000 UTC m=+647.824096792" watchObservedRunningTime="2026-11-03 20:11:57.69365198 +0000 UTC m=+647.826780442"
Nov 03 20:11:58.578736 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:58.578695    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv"]
Nov 03 20:11:58.582759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:58.582736    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv"]
Nov 03 20:11:58.582903 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:58.582833    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv"
Nov 03 20:11:58.591652 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:58.590168    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9dd42e13-serving-cert\""
Nov 03 20:11:58.715525 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:58.715495    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67abb7be-93e4-4f8a-825f-2622615f1e8d-openshift-service-ca\") pod \"maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv\" (UID: \"67abb7be-93e4-4f8a-825f-2622615f1e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv"
Nov 03 20:11:58.715702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:58.715559    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m4xr\" (UniqueName: \"kubernetes.io/projected/67abb7be-93e4-4f8a-825f-2622615f1e8d-kube-api-access-2m4xr\") pod \"maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv\" (UID: \"67abb7be-93e4-4f8a-825f-2622615f1e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv"
Nov 03 20:11:58.715702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:58.715661    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67abb7be-93e4-4f8a-825f-2622615f1e8d-maas-api-tls\") pod \"maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv\" (UID: \"67abb7be-93e4-4f8a-825f-2622615f1e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv"
Nov 03 20:11:58.715783 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:58.715707    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67abb7be-93e4-4f8a-825f-2622615f1e8d-metrics-certs\") pod \"maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv\" (UID: \"67abb7be-93e4-4f8a-825f-2622615f1e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv"
Nov 03 20:11:58.817144 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:58.817102    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m4xr\" (UniqueName: \"kubernetes.io/projected/67abb7be-93e4-4f8a-825f-2622615f1e8d-kube-api-access-2m4xr\") pod \"maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv\" (UID: \"67abb7be-93e4-4f8a-825f-2622615f1e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv"
Nov 03 20:11:58.817342 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:58.817167    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67abb7be-93e4-4f8a-825f-2622615f1e8d-maas-api-tls\") pod \"maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv\" (UID: \"67abb7be-93e4-4f8a-825f-2622615f1e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv"
Nov 03 20:11:58.817342 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:58.817224    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67abb7be-93e4-4f8a-825f-2622615f1e8d-metrics-certs\") pod \"maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv\" (UID: \"67abb7be-93e4-4f8a-825f-2622615f1e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv"
Nov 03 20:11:58.817342 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:58.817314    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67abb7be-93e4-4f8a-825f-2622615f1e8d-openshift-service-ca\") pod \"maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv\" (UID: \"67abb7be-93e4-4f8a-825f-2622615f1e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv"
Nov 03 20:11:58.818206 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:58.818179    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67abb7be-93e4-4f8a-825f-2622615f1e8d-openshift-service-ca\") pod \"maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv\" (UID: \"67abb7be-93e4-4f8a-825f-2622615f1e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv"
Nov 03 20:11:58.821026 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:58.821001    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67abb7be-93e4-4f8a-825f-2622615f1e8d-metrics-certs\") pod \"maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv\" (UID: \"67abb7be-93e4-4f8a-825f-2622615f1e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv"
Nov 03 20:11:58.827658 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:58.827631    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67abb7be-93e4-4f8a-825f-2622615f1e8d-maas-api-tls\") pod \"maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv\" (UID: \"67abb7be-93e4-4f8a-825f-2622615f1e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv"
Nov 03 20:11:58.829937 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:58.829875    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m4xr\" (UniqueName: \"kubernetes.io/projected/67abb7be-93e4-4f8a-825f-2622615f1e8d-kube-api-access-2m4xr\") pod \"maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv\" (UID: \"67abb7be-93e4-4f8a-825f-2622615f1e8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv"
Nov 03 20:11:58.900937 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:58.900893    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv"
Nov 03 20:11:59.203426 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:59.203402    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv"]
Nov 03 20:11:59.205045 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:11:59.205018    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67abb7be_93e4_4f8a_825f_2622615f1e8d.slice/crio-e0d583c1f2aeae2f9d5c945a9a3b0873a4d69f2800e9344c03df74676757b4be WatchSource:0}: Error finding container e0d583c1f2aeae2f9d5c945a9a3b0873a4d69f2800e9344c03df74676757b4be: Status 404 returned error can't find the container with id e0d583c1f2aeae2f9d5c945a9a3b0873a4d69f2800e9344c03df74676757b4be
Nov 03 20:11:59.693998 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:11:59.693935    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv" event={"ID":"67abb7be-93e4-4f8a-825f-2622615f1e8d","Type":"ContainerStarted","Data":"e0d583c1f2aeae2f9d5c945a9a3b0873a4d69f2800e9344c03df74676757b4be"}
Nov 03 20:12:00.700383 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:00.700346    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv" event={"ID":"67abb7be-93e4-4f8a-825f-2622615f1e8d","Type":"ContainerStarted","Data":"38c42b74f45ea4ce663f3cd7004fdc8836a2f41a1414be848ebf73ee7e04e4d2"}
Nov 03 20:12:00.700783 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:00.700486    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv"
Nov 03 20:12:00.723723 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:00.723675    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv" podStartSLOduration=2.313332624 podStartE2EDuration="2.723661077s" totalImagesPullingTime="410.328453ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:00.722023962 +0000 UTC m=+650.855152425" watchObservedRunningTime="2026-11-03 20:12:00.723661077 +0000 UTC m=+650.856789540"
Nov 03 20:12:01.668658 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:01.664888    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d"
Nov 03 20:12:03.199589 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:03.199560    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq"]
Nov 03 20:12:03.216629 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:03.216151    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq"]
Nov 03 20:12:03.216629 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:03.216277    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq"
Nov 03 20:12:03.225825 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:03.225800    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-2f29441c\""
Nov 03 20:12:03.226676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:03.226660    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-2f29441c-dockercfg-928vc\""
Nov 03 20:12:03.234949 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:03.234929    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2f29441c-68679b869d-kd68l"]
Nov 03 20:12:03.239307 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:03.239285    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2f29441c-68679b869d-kd68l"
Nov 03 20:12:03.248864 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:03.248816    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2f29441c-68679b869d-kd68l"]
Nov 03 20:12:03.393446 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:03.393407    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f56fq\" (UniqueName: \"kubernetes.io/projected/4c519650-430a-4012-ab7e-3eeb88567c36-kube-api-access-f56fq\") pod \"payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq\" (UID: \"4c519650-430a-4012-ab7e-3eeb88567c36\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq"
Nov 03 20:12:03.393656 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:03.393476    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c519650-430a-4012-ab7e-3eeb88567c36-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq\" (UID: \"4c519650-430a-4012-ab7e-3eeb88567c36\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq"
Nov 03 20:12:03.393656 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:03.393567    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjdq5\" (UniqueName: \"kubernetes.io/projected/e3fdb849-094e-42a1-a807-d663a6356515-kube-api-access-vjdq5\") pod \"payload-processing-e2e-derive-2f29441c-68679b869d-kd68l\" (UID: \"e3fdb849-094e-42a1-a807-d663a6356515\") " pod="openshift-ingress/payload-processing-e2e-derive-2f29441c-68679b869d-kd68l"
Nov 03 20:12:03.393656 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:03.393652    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3fdb849-094e-42a1-a807-d663a6356515-plugins-config-volume\") pod \"payload-processing-e2e-derive-2f29441c-68679b869d-kd68l\" (UID: \"e3fdb849-094e-42a1-a807-d663a6356515\") " pod="openshift-ingress/payload-processing-e2e-derive-2f29441c-68679b869d-kd68l"
Nov 03 20:12:03.494374 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:03.494283    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f56fq\" (UniqueName: \"kubernetes.io/projected/4c519650-430a-4012-ab7e-3eeb88567c36-kube-api-access-f56fq\") pod \"payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq\" (UID: \"4c519650-430a-4012-ab7e-3eeb88567c36\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq"
Nov 03 20:12:03.494374 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:03.494353    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c519650-430a-4012-ab7e-3eeb88567c36-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq\" (UID: \"4c519650-430a-4012-ab7e-3eeb88567c36\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq"
Nov 03 20:12:03.494663 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:03.494531    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjdq5\" (UniqueName: \"kubernetes.io/projected/e3fdb849-094e-42a1-a807-d663a6356515-kube-api-access-vjdq5\") pod \"payload-processing-e2e-derive-2f29441c-68679b869d-kd68l\" (UID: \"e3fdb849-094e-42a1-a807-d663a6356515\") " pod="openshift-ingress/payload-processing-e2e-derive-2f29441c-68679b869d-kd68l"
Nov 03 20:12:03.494663 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:03.494659    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3fdb849-094e-42a1-a807-d663a6356515-plugins-config-volume\") pod \"payload-processing-e2e-derive-2f29441c-68679b869d-kd68l\" (UID: \"e3fdb849-094e-42a1-a807-d663a6356515\") " pod="openshift-ingress/payload-processing-e2e-derive-2f29441c-68679b869d-kd68l"
Nov 03 20:12:03.495160 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:03.495141    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c519650-430a-4012-ab7e-3eeb88567c36-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq\" (UID: \"4c519650-430a-4012-ab7e-3eeb88567c36\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq"
Nov 03 20:12:03.495304 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:03.495285    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3fdb849-094e-42a1-a807-d663a6356515-plugins-config-volume\") pod \"payload-processing-e2e-derive-2f29441c-68679b869d-kd68l\" (UID: \"e3fdb849-094e-42a1-a807-d663a6356515\") " pod="openshift-ingress/payload-processing-e2e-derive-2f29441c-68679b869d-kd68l"
Nov 03 20:12:03.511051 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:03.511024    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f56fq\" (UniqueName: \"kubernetes.io/projected/4c519650-430a-4012-ab7e-3eeb88567c36-kube-api-access-f56fq\") pod \"payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq\" (UID: \"4c519650-430a-4012-ab7e-3eeb88567c36\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq"
Nov 03 20:12:03.511217 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:03.511193    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjdq5\" (UniqueName: \"kubernetes.io/projected/e3fdb849-094e-42a1-a807-d663a6356515-kube-api-access-vjdq5\") pod \"payload-processing-e2e-derive-2f29441c-68679b869d-kd68l\" (UID: \"e3fdb849-094e-42a1-a807-d663a6356515\") " pod="openshift-ingress/payload-processing-e2e-derive-2f29441c-68679b869d-kd68l"
Nov 03 20:12:03.532468 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:03.532438    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq"
Nov 03 20:12:03.558466 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:03.558442    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2f29441c-68679b869d-kd68l"
Nov 03 20:12:03.821064 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:03.821036    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq"]
Nov 03 20:12:03.822110 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:12:03.822072    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c519650_430a_4012_ab7e_3eeb88567c36.slice/crio-ceef58f7fc6ba0617dbb996d7c29b47b5c19ce5ec940e345932c63ad34f3529a WatchSource:0}: Error finding container ceef58f7fc6ba0617dbb996d7c29b47b5c19ce5ec940e345932c63ad34f3529a: Status 404 returned error can't find the container with id ceef58f7fc6ba0617dbb996d7c29b47b5c19ce5ec940e345932c63ad34f3529a
Nov 03 20:12:03.853991 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:03.853968    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2f29441c-68679b869d-kd68l"]
Nov 03 20:12:03.856469 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:12:03.856440    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3fdb849_094e_42a1_a807_d663a6356515.slice/crio-772f12dc47daca2b19a6d91f3011b14bf6585347cda7475f7cc122dead084e60 WatchSource:0}: Error finding container 772f12dc47daca2b19a6d91f3011b14bf6585347cda7475f7cc122dead084e60: Status 404 returned error can't find the container with id 772f12dc47daca2b19a6d91f3011b14bf6585347cda7475f7cc122dead084e60
Nov 03 20:12:04.732667 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:04.732631    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq" event={"ID":"4c519650-430a-4012-ab7e-3eeb88567c36","Type":"ContainerStarted","Data":"8062c73a8d58cad2ac9bb0996379406d7352d2be9d7de28af990ab85bb92859a"}
Nov 03 20:12:04.733090 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:04.732701    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq"
Nov 03 20:12:04.733090 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:04.732726    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq" event={"ID":"4c519650-430a-4012-ab7e-3eeb88567c36","Type":"ContainerStarted","Data":"ceef58f7fc6ba0617dbb996d7c29b47b5c19ce5ec940e345932c63ad34f3529a"}
Nov 03 20:12:04.734405 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:04.734379    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2f29441c-68679b869d-kd68l" event={"ID":"e3fdb849-094e-42a1-a807-d663a6356515","Type":"ContainerStarted","Data":"cd5058d996b51caecf539b5411019fc54d9a835b7ba0614a38eb38d4c4661fa7"}
Nov 03 20:12:04.734500 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:04.734426    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2f29441c-68679b869d-kd68l" event={"ID":"e3fdb849-094e-42a1-a807-d663a6356515","Type":"ContainerStarted","Data":"772f12dc47daca2b19a6d91f3011b14bf6585347cda7475f7cc122dead084e60"}
Nov 03 20:12:04.734500 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:04.734461    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-2f29441c-68679b869d-kd68l"
Nov 03 20:12:04.751765 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:04.751729    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq" podStartSLOduration=1.751717582 podStartE2EDuration="1.751717582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:04.750734738 +0000 UTC m=+654.883863203" watchObservedRunningTime="2026-11-03 20:12:04.751717582 +0000 UTC m=+654.884846044"
Nov 03 20:12:04.769093 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:04.769038    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-2f29441c-68679b869d-kd68l" podStartSLOduration=1.7690199519999998 podStartE2EDuration="1.769019952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:04.768741768 +0000 UTC m=+654.901870230" watchObservedRunningTime="2026-11-03 20:12:04.769019952 +0000 UTC m=+654.902148418"
Nov 03 20:12:05.808144 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:05.808106    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4"]
Nov 03 20:12:05.815013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:05.814990    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4"
Nov 03 20:12:05.822804 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:05.822727    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-2f29441c-serving-cert\""
Nov 03 20:12:05.827927 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:05.827900    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4"]
Nov 03 20:12:05.921276 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:05.921230    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5e02496-3aae-4a0e-9925-36742914626b-openshift-service-ca\") pod \"maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4\" (UID: \"b5e02496-3aae-4a0e-9925-36742914626b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4"
Nov 03 20:12:05.921433 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:05.921288    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5e02496-3aae-4a0e-9925-36742914626b-maas-api-tls\") pod \"maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4\" (UID: \"b5e02496-3aae-4a0e-9925-36742914626b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4"
Nov 03 20:12:05.921433 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:05.921363    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5e02496-3aae-4a0e-9925-36742914626b-metrics-certs\") pod \"maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4\" (UID: \"b5e02496-3aae-4a0e-9925-36742914626b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4"
Nov 03 20:12:05.921649 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:05.921622    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq2n6\" (UniqueName: \"kubernetes.io/projected/b5e02496-3aae-4a0e-9925-36742914626b-kube-api-access-pq2n6\") pod \"maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4\" (UID: \"b5e02496-3aae-4a0e-9925-36742914626b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4"
Nov 03 20:12:06.022578 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:06.022542    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq2n6\" (UniqueName: \"kubernetes.io/projected/b5e02496-3aae-4a0e-9925-36742914626b-kube-api-access-pq2n6\") pod \"maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4\" (UID: \"b5e02496-3aae-4a0e-9925-36742914626b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4"
Nov 03 20:12:06.022775 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:06.022651    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5e02496-3aae-4a0e-9925-36742914626b-openshift-service-ca\") pod \"maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4\" (UID: \"b5e02496-3aae-4a0e-9925-36742914626b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4"
Nov 03 20:12:06.022775 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:06.022680    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5e02496-3aae-4a0e-9925-36742914626b-maas-api-tls\") pod \"maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4\" (UID: \"b5e02496-3aae-4a0e-9925-36742914626b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4"
Nov 03 20:12:06.022906 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:06.022789    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5e02496-3aae-4a0e-9925-36742914626b-metrics-certs\") pod \"maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4\" (UID: \"b5e02496-3aae-4a0e-9925-36742914626b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4"
Nov 03 20:12:06.023466 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:06.023444    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5e02496-3aae-4a0e-9925-36742914626b-openshift-service-ca\") pod \"maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4\" (UID: \"b5e02496-3aae-4a0e-9925-36742914626b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4"
Nov 03 20:12:06.025374 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:06.025348    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5e02496-3aae-4a0e-9925-36742914626b-metrics-certs\") pod \"maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4\" (UID: \"b5e02496-3aae-4a0e-9925-36742914626b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4"
Nov 03 20:12:06.025886 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:06.025865    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5e02496-3aae-4a0e-9925-36742914626b-maas-api-tls\") pod \"maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4\" (UID: \"b5e02496-3aae-4a0e-9925-36742914626b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4"
Nov 03 20:12:06.039485 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:06.039446    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq2n6\" (UniqueName: \"kubernetes.io/projected/b5e02496-3aae-4a0e-9925-36742914626b-kube-api-access-pq2n6\") pod \"maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4\" (UID: \"b5e02496-3aae-4a0e-9925-36742914626b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4"
Nov 03 20:12:06.133068 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:06.133033    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4"
Nov 03 20:12:06.456780 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:06.456753    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4"]
Nov 03 20:12:06.458987 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:12:06.458954    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e02496_3aae_4a0e_9925_36742914626b.slice/crio-f86819f306aaf682a02efcc0a3cca87ee5af5efabea8a573f442215625836364 WatchSource:0}: Error finding container f86819f306aaf682a02efcc0a3cca87ee5af5efabea8a573f442215625836364: Status 404 returned error can't find the container with id f86819f306aaf682a02efcc0a3cca87ee5af5efabea8a573f442215625836364
Nov 03 20:12:06.712203 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:06.712124    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv"
Nov 03 20:12:06.750286 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:06.750245    4421 generic.go:356] "Generic (PLEG): container finished" podID="172103a4-4b03-4580-a3a9-2cd8a744b147" containerID="043ae63f76d2328d9fa44e016c445232f1d8258c034779c99acce7ec8a8e676f" exitCode=0
Nov 03 20:12:06.750589 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:06.750566    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm" event={"ID":"172103a4-4b03-4580-a3a9-2cd8a744b147","Type":"ContainerDied","Data":"043ae63f76d2328d9fa44e016c445232f1d8258c034779c99acce7ec8a8e676f"}
Nov 03 20:12:06.751364 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:06.751345    4421 scope.go:122] "RemoveContainer" containerID="043ae63f76d2328d9fa44e016c445232f1d8258c034779c99acce7ec8a8e676f"
Nov 03 20:12:06.755084 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:06.754971    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4" event={"ID":"b5e02496-3aae-4a0e-9925-36742914626b","Type":"ContainerStarted","Data":"f86819f306aaf682a02efcc0a3cca87ee5af5efabea8a573f442215625836364"}
Nov 03 20:12:06.761380 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:06.761343    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:06.763061 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:06.763036    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:06.763562 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:06.763211    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:07.796858 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:07.796818    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm" event={"ID":"172103a4-4b03-4580-a3a9-2cd8a744b147","Type":"ContainerStarted","Data":"69caa3bdda4f3da6877cf6eddda34d9ad957f3c92ba22e120b77bf369ff8cd01"}
Nov 03 20:12:07.799096 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:07.799066    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4" event={"ID":"b5e02496-3aae-4a0e-9925-36742914626b","Type":"ContainerStarted","Data":"8f0c3b3f1382a689b5abd6cff62c0dd98bd7e8d8aa00833f73935a399195a0fe"}
Nov 03 20:12:07.799252 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:07.799140    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4"
Nov 03 20:12:07.901917 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:07.900258    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:12:07.920878 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:07.916962    4421 patch_prober.go:29] interesting pod/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 20:12:07.920878 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:07.917025    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm" podUID="172103a4-4b03-4580-a3a9-2cd8a744b147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 20:12:08.617802 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:08.617767    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:12:08.729624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:08.725834    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm"
Nov 03 20:12:08.730226 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:08.730207    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth"
Nov 03 20:12:08.730753 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:08.730722    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw"
Nov 03 20:12:08.805002 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:08.804856    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4" podStartSLOduration=3.378315366 podStartE2EDuration="3.8048339s" totalImagesPullingTime="426.518534ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:07.854582158 +0000 UTC m=+657.987710622" watchObservedRunningTime="2026-11-03 20:12:08.8048339 +0000 UTC m=+658.937962363"
Nov 03 20:12:08.902085 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:08.901999    4421 patch_prober.go:29] interesting pod/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 20:12:08.902391 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:08.902359    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm" podUID="172103a4-4b03-4580-a3a9-2cd8a744b147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 20:12:09.897639 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:09.897151    4421 patch_prober.go:29] interesting pod/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 20:12:09.897639 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:09.897207    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm" podUID="172103a4-4b03-4580-a3a9-2cd8a744b147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 20:12:10.889625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:10.888673    4421 patch_prober.go:29] interesting pod/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 20:12:10.889625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:10.888745    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm" podUID="172103a4-4b03-4580-a3a9-2cd8a744b147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 20:12:11.885030 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:11.884989    4421 patch_prober.go:29] interesting pod/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 20:12:11.894816 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:11.894772    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm" podUID="172103a4-4b03-4580-a3a9-2cd8a744b147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 20:12:12.885577 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:12.885530    4421 patch_prober.go:29] interesting pod/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 20:12:12.885996 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:12.885654    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm" podUID="172103a4-4b03-4580-a3a9-2cd8a744b147" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 20:12:13.816196 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:13.816161    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4"
Nov 03 20:12:13.889868 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:13.889841    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:12:13.891235 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:13.891211    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:12:15.743437 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:15.743409    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-2f29441c-68679b869d-kd68l"
Nov 03 20:12:15.744699 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:15.743489    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq"
Nov 03 20:12:15.943565 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:15.943529    4421 generic.go:356] "Generic (PLEG): container finished" podID="ecbd165b-b0c7-436d-b1ef-6dae96349766" containerID="fceba77512b24390bbb55cedf987f2684ea797234fc9972a11576f2e046fb96d" exitCode=0
Nov 03 20:12:15.943743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:15.943592    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm" event={"ID":"ecbd165b-b0c7-436d-b1ef-6dae96349766","Type":"ContainerDied","Data":"fceba77512b24390bbb55cedf987f2684ea797234fc9972a11576f2e046fb96d"}
Nov 03 20:12:15.944031 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:15.944004    4421 scope.go:122] "RemoveContainer" containerID="fceba77512b24390bbb55cedf987f2684ea797234fc9972a11576f2e046fb96d"
Nov 03 20:12:15.945415 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:15.945385    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:15.945522 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:15.945454    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:15.945522 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:15.945488    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:16.970412 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:16.970376    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm" event={"ID":"ecbd165b-b0c7-436d-b1ef-6dae96349766","Type":"ContainerStarted","Data":"b0fa49cf5ca7b76a18bfadcfadfa401a0cdd162cbd19ac7cad6840dcb3d13836"}
Nov 03 20:12:17.466769 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:17.461014    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:12:17.466769 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:17.461141    4421 patch_prober.go:29] interesting pod/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 20:12:17.466769 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:17.461203    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm" podUID="ecbd165b-b0c7-436d-b1ef-6dae96349766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 20:12:18.465620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:18.461075    4421 patch_prober.go:29] interesting pod/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 20:12:18.465620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:18.461141    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm" podUID="ecbd165b-b0c7-436d-b1ef-6dae96349766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 20:12:19.467181 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:19.467078    4421 patch_prober.go:29] interesting pod/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 20:12:19.467181 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:19.467160    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm" podUID="ecbd165b-b0c7-436d-b1ef-6dae96349766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 20:12:19.629156 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:12:19.629118    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:12:20.463060 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:20.463020    4421 patch_prober.go:29] interesting pod/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 20:12:20.463400 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:20.463371    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm" podUID="ecbd165b-b0c7-436d-b1ef-6dae96349766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 20:12:21.465024 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:21.464990    4421 patch_prober.go:29] interesting pod/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 20:12:21.465619 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:21.465565    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm" podUID="ecbd165b-b0c7-436d-b1ef-6dae96349766" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 20:12:22.465925 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:22.465897    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:12:22.466397 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:22.466303    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:12:22.467077 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:22.467057    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:12:22.484181 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:22.484152    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d"]
Nov 03 20:12:22.484428 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:22.484380    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d" podUID="a6ae90b7-a7bf-4dc3-b132-020d7a4044e8" containerName="maas-api" containerID="cri-o://8e74a7e5a0513c714729218e260e0ac16a4e0cd1ecff7eda675fa1fe6aea8db5" gracePeriod=30
Nov 03 20:12:22.771246 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:22.771222    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d"
Nov 03 20:12:22.851653 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:22.851591    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-kube-api-access-sthfc\" (UniqueName: \"kubernetes.io/projected/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-kube-api-access-sthfc\") pod \"a6ae90b7-a7bf-4dc3-b132-020d7a4044e8\" (UID: \"a6ae90b7-a7bf-4dc3-b132-020d7a4044e8\") "
Nov 03 20:12:22.851838 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:22.851705    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-openshift-service-ca\") pod \"a6ae90b7-a7bf-4dc3-b132-020d7a4044e8\" (UID: \"a6ae90b7-a7bf-4dc3-b132-020d7a4044e8\") "
Nov 03 20:12:22.851838 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:22.851759    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-maas-api-tls\") pod \"a6ae90b7-a7bf-4dc3-b132-020d7a4044e8\" (UID: \"a6ae90b7-a7bf-4dc3-b132-020d7a4044e8\") "
Nov 03 20:12:22.851838 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:22.851791    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-metrics-certs\") pod \"a6ae90b7-a7bf-4dc3-b132-020d7a4044e8\" (UID: \"a6ae90b7-a7bf-4dc3-b132-020d7a4044e8\") "
Nov 03 20:12:22.852166 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:22.852128    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-openshift-service-ca" pod "a6ae90b7-a7bf-4dc3-b132-020d7a4044e8" (UID: "a6ae90b7-a7bf-4dc3-b132-020d7a4044e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:22.852437 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:22.852417    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:22.854113 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:22.854085    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-kube-api-access-sthfc" pod "a6ae90b7-a7bf-4dc3-b132-020d7a4044e8" (UID: "a6ae90b7-a7bf-4dc3-b132-020d7a4044e8"). InnerVolumeSpecName "kube-api-access-sthfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:22.854493 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:22.854466    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-maas-api-tls" pod "a6ae90b7-a7bf-4dc3-b132-020d7a4044e8" (UID: "a6ae90b7-a7bf-4dc3-b132-020d7a4044e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:12:22.854493 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:22.854481    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-metrics-certs" pod "a6ae90b7-a7bf-4dc3-b132-020d7a4044e8" (UID: "a6ae90b7-a7bf-4dc3-b132-020d7a4044e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:12:22.953655 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:22.953622    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sthfc\" (UniqueName: \"kubernetes.io/projected/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-kube-api-access-sthfc\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:22.953655 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:22.953656    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:22.953874 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:22.953671    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:22.980672 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:22.980637    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"]
Nov 03 20:12:22.980914 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:22.980885    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm" podUID="172103a4-4b03-4580-a3a9-2cd8a744b147" containerName="istio-proxy" containerID="cri-o://69caa3bdda4f3da6877cf6eddda34d9ad957f3c92ba22e120b77bf369ff8cd01" gracePeriod=30
Nov 03 20:12:23.009805 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:23.009715    4421 generic.go:356] "Generic (PLEG): container finished" podID="a6ae90b7-a7bf-4dc3-b132-020d7a4044e8" containerID="8e74a7e5a0513c714729218e260e0ac16a4e0cd1ecff7eda675fa1fe6aea8db5" exitCode=0
Nov 03 20:12:23.009805 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:23.009772    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d"
Nov 03 20:12:23.010016 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:23.009796    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d" event={"ID":"a6ae90b7-a7bf-4dc3-b132-020d7a4044e8","Type":"ContainerDied","Data":"8e74a7e5a0513c714729218e260e0ac16a4e0cd1ecff7eda675fa1fe6aea8db5"}
Nov 03 20:12:23.010016 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:23.009874    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d" event={"ID":"a6ae90b7-a7bf-4dc3-b132-020d7a4044e8","Type":"ContainerDied","Data":"97581bf4491b4e28a60bf0b5adc7ab4e8f071f4018ae56c4035580e968df8df9"}
Nov 03 20:12:23.010016 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:23.009910    4421 scope.go:122] "RemoveContainer" containerID="8e74a7e5a0513c714729218e260e0ac16a4e0cd1ecff7eda675fa1fe6aea8db5"
Nov 03 20:12:23.019882 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:23.019838    4421 scope.go:122] "RemoveContainer" containerID="8e74a7e5a0513c714729218e260e0ac16a4e0cd1ecff7eda675fa1fe6aea8db5"
Nov 03 20:12:23.020130 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:12:23.020112    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e74a7e5a0513c714729218e260e0ac16a4e0cd1ecff7eda675fa1fe6aea8db5\": container with ID starting with 8e74a7e5a0513c714729218e260e0ac16a4e0cd1ecff7eda675fa1fe6aea8db5 not found: ID does not exist" containerID="8e74a7e5a0513c714729218e260e0ac16a4e0cd1ecff7eda675fa1fe6aea8db5"
Nov 03 20:12:23.020188 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:23.020140    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e74a7e5a0513c714729218e260e0ac16a4e0cd1ecff7eda675fa1fe6aea8db5"} err="failed to get container status \"8e74a7e5a0513c714729218e260e0ac16a4e0cd1ecff7eda675fa1fe6aea8db5\": rpc error: code = NotFound desc = could not find container \"8e74a7e5a0513c714729218e260e0ac16a4e0cd1ecff7eda675fa1fe6aea8db5\": container with ID starting with 8e74a7e5a0513c714729218e260e0ac16a4e0cd1ecff7eda675fa1fe6aea8db5 not found: ID does not exist"
Nov 03 20:12:23.032727 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:23.032701    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d"]
Nov 03 20:12:23.039991 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:23.039972    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-459ca04a-5885b88747-ld96d"]
Nov 03 20:12:24.019908 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:24.019870    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6ae90b7-a7bf-4dc3-b132-020d7a4044e8" path="/var/lib/kubelet/pods/a6ae90b7-a7bf-4dc3-b132-020d7a4044e8/volumes"
Nov 03 20:12:24.566503 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:24.566450    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw"]
Nov 03 20:12:24.775222 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:24.775186    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth"]
Nov 03 20:12:24.818379 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:24.808882    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv"]
Nov 03 20:12:24.818379 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:24.809150    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv" podUID="67abb7be-93e4-4f8a-825f-2622615f1e8d" containerName="maas-api" containerID="cri-o://38c42b74f45ea4ce663f3cd7004fdc8836a2f41a1414be848ebf73ee7e04e4d2" gracePeriod=30
Nov 03 20:12:25.031536 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:25.031505    4421 generic.go:356] "Generic (PLEG): container finished" podID="67abb7be-93e4-4f8a-825f-2622615f1e8d" containerID="38c42b74f45ea4ce663f3cd7004fdc8836a2f41a1414be848ebf73ee7e04e4d2" exitCode=0
Nov 03 20:12:25.032029 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:25.031542    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv" event={"ID":"67abb7be-93e4-4f8a-825f-2622615f1e8d","Type":"ContainerDied","Data":"38c42b74f45ea4ce663f3cd7004fdc8836a2f41a1414be848ebf73ee7e04e4d2"}
Nov 03 20:12:25.065487 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:25.065467    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv"
Nov 03 20:12:25.188740 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:25.188707    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67abb7be-93e4-4f8a-825f-2622615f1e8d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67abb7be-93e4-4f8a-825f-2622615f1e8d-maas-api-tls\") pod \"67abb7be-93e4-4f8a-825f-2622615f1e8d\" (UID: \"67abb7be-93e4-4f8a-825f-2622615f1e8d\") "
Nov 03 20:12:25.188917 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:25.188750    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67abb7be-93e4-4f8a-825f-2622615f1e8d-kube-api-access-2m4xr\" (UniqueName: \"kubernetes.io/projected/67abb7be-93e4-4f8a-825f-2622615f1e8d-kube-api-access-2m4xr\") pod \"67abb7be-93e4-4f8a-825f-2622615f1e8d\" (UID: \"67abb7be-93e4-4f8a-825f-2622615f1e8d\") "
Nov 03 20:12:25.188917 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:25.188790    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67abb7be-93e4-4f8a-825f-2622615f1e8d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67abb7be-93e4-4f8a-825f-2622615f1e8d-openshift-service-ca\") pod \"67abb7be-93e4-4f8a-825f-2622615f1e8d\" (UID: \"67abb7be-93e4-4f8a-825f-2622615f1e8d\") "
Nov 03 20:12:25.188917 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:25.188888    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67abb7be-93e4-4f8a-825f-2622615f1e8d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/67abb7be-93e4-4f8a-825f-2622615f1e8d-metrics-certs\") pod \"67abb7be-93e4-4f8a-825f-2622615f1e8d\" (UID: \"67abb7be-93e4-4f8a-825f-2622615f1e8d\") "
Nov 03 20:12:25.189255 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:25.189229    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67abb7be-93e4-4f8a-825f-2622615f1e8d-openshift-service-ca" pod "67abb7be-93e4-4f8a-825f-2622615f1e8d" (UID: "67abb7be-93e4-4f8a-825f-2622615f1e8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:25.191109 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:25.191082    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67abb7be-93e4-4f8a-825f-2622615f1e8d-maas-api-tls" pod "67abb7be-93e4-4f8a-825f-2622615f1e8d" (UID: "67abb7be-93e4-4f8a-825f-2622615f1e8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:12:25.191286 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:25.191271    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67abb7be-93e4-4f8a-825f-2622615f1e8d-kube-api-access-2m4xr" pod "67abb7be-93e4-4f8a-825f-2622615f1e8d" (UID: "67abb7be-93e4-4f8a-825f-2622615f1e8d"). InnerVolumeSpecName "kube-api-access-2m4xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:25.191432 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:25.191410    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67abb7be-93e4-4f8a-825f-2622615f1e8d-metrics-certs" pod "67abb7be-93e4-4f8a-825f-2622615f1e8d" (UID: "67abb7be-93e4-4f8a-825f-2622615f1e8d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:12:25.296375 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:25.296270    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67abb7be-93e4-4f8a-825f-2622615f1e8d-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:25.296375 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:25.296307    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67abb7be-93e4-4f8a-825f-2622615f1e8d-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:25.296375 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:25.296321    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m4xr\" (UniqueName: \"kubernetes.io/projected/67abb7be-93e4-4f8a-825f-2622615f1e8d-kube-api-access-2m4xr\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:25.296375 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:25.296334    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67abb7be-93e4-4f8a-825f-2622615f1e8d-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:26.044075 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:26.044046    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv"
Nov 03 20:12:26.044486 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:26.044049    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv" event={"ID":"67abb7be-93e4-4f8a-825f-2622615f1e8d","Type":"ContainerDied","Data":"e0d583c1f2aeae2f9d5c945a9a3b0873a4d69f2800e9344c03df74676757b4be"}
Nov 03 20:12:26.044567 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:26.044512    4421 scope.go:122] "RemoveContainer" containerID="38c42b74f45ea4ce663f3cd7004fdc8836a2f41a1414be848ebf73ee7e04e4d2"
Nov 03 20:12:26.063223 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:26.063198    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv"]
Nov 03 20:12:26.069163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:26.069138    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9dd42e13-86d64bf689-sg4dv"]
Nov 03 20:12:26.268848 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:26.268817    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm"]
Nov 03 20:12:26.803973 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:26.803938    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"]
Nov 03 20:12:26.804166 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:26.804134    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm" podUID="ecbd165b-b0c7-436d-b1ef-6dae96349766" containerName="istio-proxy" containerID="cri-o://b0fa49cf5ca7b76a18bfadcfadfa401a0cdd162cbd19ac7cad6840dcb3d13836" gracePeriod=30
Nov 03 20:12:28.015382 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.015350    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67abb7be-93e4-4f8a-825f-2622615f1e8d" path="/var/lib/kubelet/pods/67abb7be-93e4-4f8a-825f-2622615f1e8d/volumes"
Nov 03 20:12:28.327254 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.326972    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:12:28.463021 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.462989    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-workload-certs\") pod \"172103a4-4b03-4580-a3a9-2cd8a744b147\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") "
Nov 03 20:12:28.463021 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.463025    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-token\" (UniqueName: \"kubernetes.io/projected/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-token\") pod \"172103a4-4b03-4580-a3a9-2cd8a744b147\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") "
Nov 03 20:12:28.463247 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.463067    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/172103a4-4b03-4580-a3a9-2cd8a744b147-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/172103a4-4b03-4580-a3a9-2cd8a744b147-istiod-ca-cert\") pod \"172103a4-4b03-4580-a3a9-2cd8a744b147\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") "
Nov 03 20:12:28.463247 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.463095    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-podinfo\") pod \"172103a4-4b03-4580-a3a9-2cd8a744b147\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") "
Nov 03 20:12:28.463247 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.463155    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-credential-socket\") pod \"172103a4-4b03-4580-a3a9-2cd8a744b147\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") "
Nov 03 20:12:28.463247 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.463207    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-envoy\") pod \"172103a4-4b03-4580-a3a9-2cd8a744b147\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") "
Nov 03 20:12:28.463451 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.463258    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-data\") pod \"172103a4-4b03-4580-a3a9-2cd8a744b147\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") "
Nov 03 20:12:28.463451 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.463298    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-workload-certs" pod "172103a4-4b03-4580-a3a9-2cd8a744b147" (UID: "172103a4-4b03-4580-a3a9-2cd8a744b147"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:28.463451 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.463397    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/172103a4-4b03-4580-a3a9-2cd8a744b147-kube-api-access-42rrt\" (UniqueName: \"kubernetes.io/projected/172103a4-4b03-4580-a3a9-2cd8a744b147-kube-api-access-42rrt\") pod \"172103a4-4b03-4580-a3a9-2cd8a744b147\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") "
Nov 03 20:12:28.463451 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.463428    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-workload-socket\") pod \"172103a4-4b03-4580-a3a9-2cd8a744b147\" (UID: \"172103a4-4b03-4580-a3a9-2cd8a744b147\") "
Nov 03 20:12:28.463727 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.463686    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-credential-socket" pod "172103a4-4b03-4580-a3a9-2cd8a744b147" (UID: "172103a4-4b03-4580-a3a9-2cd8a744b147"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:28.464163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.463837    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/172103a4-4b03-4580-a3a9-2cd8a744b147-istiod-ca-cert" pod "172103a4-4b03-4580-a3a9-2cd8a744b147" (UID: "172103a4-4b03-4580-a3a9-2cd8a744b147"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:28.464163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.464029    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-workload-socket" pod "172103a4-4b03-4580-a3a9-2cd8a744b147" (UID: "172103a4-4b03-4580-a3a9-2cd8a744b147"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:28.464163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.464068    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-data" pod "172103a4-4b03-4580-a3a9-2cd8a744b147" (UID: "172103a4-4b03-4580-a3a9-2cd8a744b147"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:28.464475 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.464348    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:28.464475 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.464371    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:28.464475 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.464387    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:28.464475 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.464401    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/172103a4-4b03-4580-a3a9-2cd8a744b147-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:28.464475 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.464415    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:28.465910 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.465585    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-token" pod "172103a4-4b03-4580-a3a9-2cd8a744b147" (UID: "172103a4-4b03-4580-a3a9-2cd8a744b147"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:28.466330 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.466306    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-envoy" pod "172103a4-4b03-4580-a3a9-2cd8a744b147" (UID: "172103a4-4b03-4580-a3a9-2cd8a744b147"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:28.466629 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.466532    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-podinfo" pod "172103a4-4b03-4580-a3a9-2cd8a744b147" (UID: "172103a4-4b03-4580-a3a9-2cd8a744b147"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:12:28.466720 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.466672    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/172103a4-4b03-4580-a3a9-2cd8a744b147-kube-api-access-42rrt" pod "172103a4-4b03-4580-a3a9-2cd8a744b147" (UID: "172103a4-4b03-4580-a3a9-2cd8a744b147"). InnerVolumeSpecName "kube-api-access-42rrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:28.565658 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.565549    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42rrt\" (UniqueName: \"kubernetes.io/projected/172103a4-4b03-4580-a3a9-2cd8a744b147-kube-api-access-42rrt\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:28.565658 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.565579    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:28.565658 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.565588    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:28.565658 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.565622    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/172103a4-4b03-4580-a3a9-2cd8a744b147-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:28.891106 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.891019    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"]
Nov 03 20:12:28.891930 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.891905    4421 cpu_manager.go:414] "containerMap: removing container" podUID="67abb7be-93e4-4f8a-825f-2622615f1e8d" containerName="maas-api"
Nov 03 20:12:28.891930 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.891931    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67abb7be-93e4-4f8a-825f-2622615f1e8d" containerName="maas-api"
Nov 03 20:12:28.892079 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.891968    4421 cpu_manager.go:414] "containerMap: removing container" podUID="172103a4-4b03-4580-a3a9-2cd8a744b147" containerName="istio-proxy"
Nov 03 20:12:28.892079 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.891973    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="172103a4-4b03-4580-a3a9-2cd8a744b147" containerName="istio-proxy"
Nov 03 20:12:28.892079 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.891984    4421 cpu_manager.go:414] "containerMap: removing container" podUID="172103a4-4b03-4580-a3a9-2cd8a744b147" containerName="istio-proxy"
Nov 03 20:12:28.892079 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.891989    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="172103a4-4b03-4580-a3a9-2cd8a744b147" containerName="istio-proxy"
Nov 03 20:12:28.892079 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.891999    4421 cpu_manager.go:414] "containerMap: removing container" podUID="a6ae90b7-a7bf-4dc3-b132-020d7a4044e8" containerName="maas-api"
Nov 03 20:12:28.892079 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.892007    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6ae90b7-a7bf-4dc3-b132-020d7a4044e8" containerName="maas-api"
Nov 03 20:12:28.892253 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.892086    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="172103a4-4b03-4580-a3a9-2cd8a744b147" containerName="istio-proxy"
Nov 03 20:12:28.892253 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.892098    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="172103a4-4b03-4580-a3a9-2cd8a744b147" containerName="istio-proxy"
Nov 03 20:12:28.892253 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.892105    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6ae90b7-a7bf-4dc3-b132-020d7a4044e8" containerName="maas-api"
Nov 03 20:12:28.892253 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.892113    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="67abb7be-93e4-4f8a-825f-2622615f1e8d" containerName="maas-api"
Nov 03 20:12:28.895456 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.895430    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:28.900051 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.900028    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-085566-openshift-default-dockercfg-dt25g\""
Nov 03 20:12:28.924029 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:28.923997    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"]
Nov 03 20:12:29.066987 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.066945    4421 generic.go:356] "Generic (PLEG): container finished" podID="172103a4-4b03-4580-a3a9-2cd8a744b147" containerID="69caa3bdda4f3da6877cf6eddda34d9ad957f3c92ba22e120b77bf369ff8cd01" exitCode=0
Nov 03 20:12:29.067497 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.067017    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm" event={"ID":"172103a4-4b03-4580-a3a9-2cd8a744b147","Type":"ContainerDied","Data":"69caa3bdda4f3da6877cf6eddda34d9ad957f3c92ba22e120b77bf369ff8cd01"}
Nov 03 20:12:29.067497 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.067056    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"
Nov 03 20:12:29.067497 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.067100    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm" event={"ID":"172103a4-4b03-4580-a3a9-2cd8a744b147","Type":"ContainerDied","Data":"1ec08a1aa91ad15d17e5786d7d358adec4949292e65a33446edaf14548caa595"}
Nov 03 20:12:29.067497 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.067129    4421 scope.go:122] "RemoveContainer" containerID="69caa3bdda4f3da6877cf6eddda34d9ad957f3c92ba22e120b77bf369ff8cd01"
Nov 03 20:12:29.070677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.070367    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/830b16e0-459d-444c-b7da-c8d6d8576d4f-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.070677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.070438    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8hqw\" (UniqueName: \"kubernetes.io/projected/830b16e0-459d-444c-b7da-c8d6d8576d4f-kube-api-access-x8hqw\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.070677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.070470    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.070677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.070508    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.070677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.070535    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.070677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.070626    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.071193 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.071164    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.071376 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.071237    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.071376 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.071276    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.099431 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.099406    4421 scope.go:122] "RemoveContainer" containerID="043ae63f76d2328d9fa44e016c445232f1d8258c034779c99acce7ec8a8e676f"
Nov 03 20:12:29.112577 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.112349    4421 scope.go:122] "RemoveContainer" containerID="69caa3bdda4f3da6877cf6eddda34d9ad957f3c92ba22e120b77bf369ff8cd01"
Nov 03 20:12:29.114072 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:12:29.113069    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69caa3bdda4f3da6877cf6eddda34d9ad957f3c92ba22e120b77bf369ff8cd01\": container with ID starting with 69caa3bdda4f3da6877cf6eddda34d9ad957f3c92ba22e120b77bf369ff8cd01 not found: ID does not exist" containerID="69caa3bdda4f3da6877cf6eddda34d9ad957f3c92ba22e120b77bf369ff8cd01"
Nov 03 20:12:29.114072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.113120    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69caa3bdda4f3da6877cf6eddda34d9ad957f3c92ba22e120b77bf369ff8cd01"} err="failed to get container status \"69caa3bdda4f3da6877cf6eddda34d9ad957f3c92ba22e120b77bf369ff8cd01\": rpc error: code = NotFound desc = could not find container \"69caa3bdda4f3da6877cf6eddda34d9ad957f3c92ba22e120b77bf369ff8cd01\": container with ID starting with 69caa3bdda4f3da6877cf6eddda34d9ad957f3c92ba22e120b77bf369ff8cd01 not found: ID does not exist"
Nov 03 20:12:29.114072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.113144    4421 scope.go:122] "RemoveContainer" containerID="043ae63f76d2328d9fa44e016c445232f1d8258c034779c99acce7ec8a8e676f"
Nov 03 20:12:29.114072 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:12:29.113498    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"043ae63f76d2328d9fa44e016c445232f1d8258c034779c99acce7ec8a8e676f\": container with ID starting with 043ae63f76d2328d9fa44e016c445232f1d8258c034779c99acce7ec8a8e676f not found: ID does not exist" containerID="043ae63f76d2328d9fa44e016c445232f1d8258c034779c99acce7ec8a8e676f"
Nov 03 20:12:29.114072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.113549    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"043ae63f76d2328d9fa44e016c445232f1d8258c034779c99acce7ec8a8e676f"} err="failed to get container status \"043ae63f76d2328d9fa44e016c445232f1d8258c034779c99acce7ec8a8e676f\": rpc error: code = NotFound desc = could not find container \"043ae63f76d2328d9fa44e016c445232f1d8258c034779c99acce7ec8a8e676f\": container with ID starting with 043ae63f76d2328d9fa44e016c445232f1d8258c034779c99acce7ec8a8e676f not found: ID does not exist"
Nov 03 20:12:29.119326 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.119303    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"]
Nov 03 20:12:29.122704 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.122673    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-459ca04a-gw-openshift-default-68b84f665c-zxrpm"]
Nov 03 20:12:29.178231 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.178143    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.178231 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.178199    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.178442 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.178272    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.178442 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.178314    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.178442 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.178384    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.178442 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.178424    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.178814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.178458    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/830b16e0-459d-444c-b7da-c8d6d8576d4f-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.178814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.178526    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8hqw\" (UniqueName: \"kubernetes.io/projected/830b16e0-459d-444c-b7da-c8d6d8576d4f-kube-api-access-x8hqw\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.178814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.178559    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.179168 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.179140    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"]
Nov 03 20:12:29.180272 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.180247    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.180374 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.180280    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.180847 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.180824    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.181877 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.181856    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/830b16e0-459d-444c-b7da-c8d6d8576d4f-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.182009 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.181996    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.184848 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.184375    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.191560 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.187681    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.191560 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.188137    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.191560 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.191159    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"]
Nov 03 20:12:29.194382 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.194325    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f3631f-openshift-default-dockercfg-2vnzz\""
Nov 03 20:12:29.208188 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.208162    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.214585 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.214558    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8hqw\" (UniqueName: \"kubernetes.io/projected/830b16e0-459d-444c-b7da-c8d6d8576d4f-kube-api-access-x8hqw\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.379904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.379869    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.380083 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.379932    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.380083 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.380029    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.380083 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.380067    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.380206 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.380106    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.380206 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.380167    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.380286 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.380206    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.380286 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.380279    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msp7k\" (UniqueName: \"kubernetes.io/projected/317e6b68-68f8-4220-a3ef-2f0999a19ad9-kube-api-access-msp7k\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.380395 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.380326    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.481427 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.481392    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.481586 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.481436    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.481586 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.481455    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.481586 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.481573    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.481779 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.481650    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.481779 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.481715    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.481779 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.481764    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.481946 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.481864    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.481946 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.481898    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msp7k\" (UniqueName: \"kubernetes.io/projected/317e6b68-68f8-4220-a3ef-2f0999a19ad9-kube-api-access-msp7k\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.482081 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.481967    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.482081 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.481973    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.482258 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.482230    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.482321 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.482243    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.482490 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.482468    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.483966 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.483942    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.484215 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.484198    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.496982 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.496955    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.497965 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.497943    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msp7k\" (UniqueName: \"kubernetes.io/projected/317e6b68-68f8-4220-a3ef-2f0999a19ad9-kube-api-access-msp7k\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.509657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.509638    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:29.522715 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.522696    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc"]
Nov 03 20:12:29.526218 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.526203    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc"
Nov 03 20:12:29.536209 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.536181    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc"]
Nov 03 20:12:29.541149 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.541130    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:29.568909 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.568676    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw" podUID="981a2ed3-5af9-45ff-b165-283348d9942e" containerName="payload-pre-processing" containerID="cri-o://357133aaaf35ab2c5fe4092fdea75c5a5bd5ca8370705f1d2b4c1c0d6c3d925f" gracePeriod=25
Nov 03 20:12:29.600660 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.594141    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-4f93d1-dockercfg-bmvch\""
Nov 03 20:12:29.600660 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.594271    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-4f93d1\""
Nov 03 20:12:29.600660 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.595193    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wjsr\" (UniqueName: \"kubernetes.io/projected/feb6a504-7873-440d-a013-36654e702307-kube-api-access-2wjsr\") pod \"payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc\" (UID: \"feb6a504-7873-440d-a013-36654e702307\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc"
Nov 03 20:12:29.600660 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.595292    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb6a504-7873-440d-a013-36654e702307-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc\" (UID: \"feb6a504-7873-440d-a013-36654e702307\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc"
Nov 03 20:12:29.696463 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.696431    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wjsr\" (UniqueName: \"kubernetes.io/projected/feb6a504-7873-440d-a013-36654e702307-kube-api-access-2wjsr\") pod \"payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc\" (UID: \"feb6a504-7873-440d-a013-36654e702307\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc"
Nov 03 20:12:29.696640 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.696537    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb6a504-7873-440d-a013-36654e702307-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc\" (UID: \"feb6a504-7873-440d-a013-36654e702307\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc"
Nov 03 20:12:29.697230 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.697208    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb6a504-7873-440d-a013-36654e702307-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc\" (UID: \"feb6a504-7873-440d-a013-36654e702307\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc"
Nov 03 20:12:29.709124 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.709091    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wjsr\" (UniqueName: \"kubernetes.io/projected/feb6a504-7873-440d-a013-36654e702307-kube-api-access-2wjsr\") pod \"payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc\" (UID: \"feb6a504-7873-440d-a013-36654e702307\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc"
Nov 03 20:12:29.775850 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.775784    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth" podUID="d19980a3-d265-46e3-be8d-88968dd205fc" containerName="payload-processing" containerID="cri-o://f124aa3695f2fa69196ad21e8814480dbd00b7382b8333dad424f91705ceaec1" gracePeriod=25
Nov 03 20:12:29.844372 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.844330    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc"
Nov 03 20:12:29.851933 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.851902    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"]
Nov 03 20:12:29.855587 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:12:29.855559    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod830b16e0_459d_444c_b7da_c8d6d8576d4f.slice/crio-74ec5585995d88ac5403097f7f672fb5a24e2f1d00dddac3994c531e452080aa WatchSource:0}: Error finding container 74ec5585995d88ac5403097f7f672fb5a24e2f1d00dddac3994c531e452080aa: Status 404 returned error can't find the container with id 74ec5585995d88ac5403097f7f672fb5a24e2f1d00dddac3994c531e452080aa
Nov 03 20:12:29.876271 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.876213    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"]
Nov 03 20:12:29.878791 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:12:29.878758    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod317e6b68_68f8_4220_a3ef_2f0999a19ad9.slice/crio-c8e1e37513ed3875cc695de89ef18758eac04f97f0722c92aacf025e6a42809a WatchSource:0}: Error finding container c8e1e37513ed3875cc695de89ef18758eac04f97f0722c92aacf025e6a42809a: Status 404 returned error can't find the container with id c8e1e37513ed3875cc695de89ef18758eac04f97f0722c92aacf025e6a42809a
Nov 03 20:12:29.884091 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.884055    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:29.884161 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.884123    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:29.884206 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.884158    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:29.884280 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.884236    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:29.884352 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.884333    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:29.884411 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.884395    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:29.894533 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.894507    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw"
Nov 03 20:12:29.906034 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.902368    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/981a2ed3-5af9-45ff-b165-283348d9942e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981a2ed3-5af9-45ff-b165-283348d9942e-plugins-config-volume\") pod \"981a2ed3-5af9-45ff-b165-283348d9942e\" (UID: \"981a2ed3-5af9-45ff-b165-283348d9942e\") "
Nov 03 20:12:29.906034 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.902473    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/981a2ed3-5af9-45ff-b165-283348d9942e-kube-api-access-9fbfg\" (UniqueName: \"kubernetes.io/projected/981a2ed3-5af9-45ff-b165-283348d9942e-kube-api-access-9fbfg\") pod \"981a2ed3-5af9-45ff-b165-283348d9942e\" (UID: \"981a2ed3-5af9-45ff-b165-283348d9942e\") "
Nov 03 20:12:29.906034 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.903230    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981a2ed3-5af9-45ff-b165-283348d9942e-plugins-config-volume" pod "981a2ed3-5af9-45ff-b165-283348d9942e" (UID: "981a2ed3-5af9-45ff-b165-283348d9942e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:29.906034 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:29.906003    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/981a2ed3-5af9-45ff-b165-283348d9942e-kube-api-access-9fbfg" pod "981a2ed3-5af9-45ff-b165-283348d9942e" (UID: "981a2ed3-5af9-45ff-b165-283348d9942e"). InnerVolumeSpecName "kube-api-access-9fbfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:30.003751 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.003708    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981a2ed3-5af9-45ff-b165-283348d9942e-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:30.003751 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.003729    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fbfg\" (UniqueName: \"kubernetes.io/projected/981a2ed3-5af9-45ff-b165-283348d9942e-kube-api-access-9fbfg\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:30.026912 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.026836    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="172103a4-4b03-4580-a3a9-2cd8a744b147" path="/var/lib/kubelet/pods/172103a4-4b03-4580-a3a9-2cd8a744b147/volumes"
Nov 03 20:12:30.102623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.093143    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4" event={"ID":"830b16e0-459d-444c-b7da-c8d6d8576d4f","Type":"ContainerStarted","Data":"296173dacd6b5ac0ad08d8f5cb4c6356ebbde4afc2a09ebf492e25eb42971f2b"}
Nov 03 20:12:30.102623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.093205    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4" event={"ID":"830b16e0-459d-444c-b7da-c8d6d8576d4f","Type":"ContainerStarted","Data":"74ec5585995d88ac5403097f7f672fb5a24e2f1d00dddac3994c531e452080aa"}
Nov 03 20:12:30.163073 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.162114    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw" event={"ID":"317e6b68-68f8-4220-a3ef-2f0999a19ad9","Type":"ContainerStarted","Data":"759d180f655772ea7fe565676cfd10e5575a2ae5a37aa2bb9454a3c57baee927"}
Nov 03 20:12:30.163073 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.162181    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw" event={"ID":"317e6b68-68f8-4220-a3ef-2f0999a19ad9","Type":"ContainerStarted","Data":"c8e1e37513ed3875cc695de89ef18758eac04f97f0722c92aacf025e6a42809a"}
Nov 03 20:12:30.183280 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.181391    4421 generic.go:356] "Generic (PLEG): container finished" podID="981a2ed3-5af9-45ff-b165-283348d9942e" containerID="357133aaaf35ab2c5fe4092fdea75c5a5bd5ca8370705f1d2b4c1c0d6c3d925f" exitCode=0
Nov 03 20:12:30.183280 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.181399    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4" podStartSLOduration=2.181380506 podStartE2EDuration="2.181380506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:30.142498579 +0000 UTC m=+680.275627041" watchObservedRunningTime="2026-11-03 20:12:30.181380506 +0000 UTC m=+680.314508969"
Nov 03 20:12:30.183280 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.181491    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw" event={"ID":"981a2ed3-5af9-45ff-b165-283348d9942e","Type":"ContainerDied","Data":"357133aaaf35ab2c5fe4092fdea75c5a5bd5ca8370705f1d2b4c1c0d6c3d925f"}
Nov 03 20:12:30.183280 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.181538    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw" event={"ID":"981a2ed3-5af9-45ff-b165-283348d9942e","Type":"ContainerDied","Data":"75029bf37fde3c0e6c37f39bbd0e8ad850e26805d5dea9f6938e16d48f01b859"}
Nov 03 20:12:30.183280 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.181566    4421 scope.go:122] "RemoveContainer" containerID="357133aaaf35ab2c5fe4092fdea75c5a5bd5ca8370705f1d2b4c1c0d6c3d925f"
Nov 03 20:12:30.183280 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.181731    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw"
Nov 03 20:12:30.183280 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.181931    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc"]
Nov 03 20:12:30.188165 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.188136    4421 generic.go:356] "Generic (PLEG): container finished" podID="d19980a3-d265-46e3-be8d-88968dd205fc" containerID="f124aa3695f2fa69196ad21e8814480dbd00b7382b8333dad424f91705ceaec1" exitCode=0
Nov 03 20:12:30.188292 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.188178    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth" event={"ID":"d19980a3-d265-46e3-be8d-88968dd205fc","Type":"ContainerDied","Data":"f124aa3695f2fa69196ad21e8814480dbd00b7382b8333dad424f91705ceaec1"}
Nov 03 20:12:30.244063 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.244043    4421 scope.go:122] "RemoveContainer" containerID="357133aaaf35ab2c5fe4092fdea75c5a5bd5ca8370705f1d2b4c1c0d6c3d925f"
Nov 03 20:12:30.244564 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:12:30.244495    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"357133aaaf35ab2c5fe4092fdea75c5a5bd5ca8370705f1d2b4c1c0d6c3d925f\": container with ID starting with 357133aaaf35ab2c5fe4092fdea75c5a5bd5ca8370705f1d2b4c1c0d6c3d925f not found: ID does not exist" containerID="357133aaaf35ab2c5fe4092fdea75c5a5bd5ca8370705f1d2b4c1c0d6c3d925f"
Nov 03 20:12:30.244564 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.244529    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"357133aaaf35ab2c5fe4092fdea75c5a5bd5ca8370705f1d2b4c1c0d6c3d925f"} err="failed to get container status \"357133aaaf35ab2c5fe4092fdea75c5a5bd5ca8370705f1d2b4c1c0d6c3d925f\": rpc error: code = NotFound desc = could not find container \"357133aaaf35ab2c5fe4092fdea75c5a5bd5ca8370705f1d2b4c1c0d6c3d925f\": container with ID starting with 357133aaaf35ab2c5fe4092fdea75c5a5bd5ca8370705f1d2b4c1c0d6c3d925f not found: ID does not exist"
Nov 03 20:12:30.245761 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.245582    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth"
Nov 03 20:12:30.256179 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.256134    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw" podStartSLOduration=1.256121962 podStartE2EDuration="1.256121962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:30.20418263 +0000 UTC m=+680.337311071" watchObservedRunningTime="2026-11-03 20:12:30.256121962 +0000 UTC m=+680.389250424"
Nov 03 20:12:30.259638 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.259614    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw"]
Nov 03 20:12:30.263437 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.263415    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e17fc195-7c5bb5666c-4bzqw"]
Nov 03 20:12:30.430028 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.429995    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d19980a3-d265-46e3-be8d-88968dd205fc-kube-api-access-pk2t7\" (UniqueName: \"kubernetes.io/projected/d19980a3-d265-46e3-be8d-88968dd205fc-kube-api-access-pk2t7\") pod \"d19980a3-d265-46e3-be8d-88968dd205fc\" (UID: \"d19980a3-d265-46e3-be8d-88968dd205fc\") "
Nov 03 20:12:30.430280 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.430258    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d19980a3-d265-46e3-be8d-88968dd205fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19980a3-d265-46e3-be8d-88968dd205fc-plugins-config-volume\") pod \"d19980a3-d265-46e3-be8d-88968dd205fc\" (UID: \"d19980a3-d265-46e3-be8d-88968dd205fc\") "
Nov 03 20:12:30.431259 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.431010    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d19980a3-d265-46e3-be8d-88968dd205fc-plugins-config-volume" pod "d19980a3-d265-46e3-be8d-88968dd205fc" (UID: "d19980a3-d265-46e3-be8d-88968dd205fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:30.433382 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.433353    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d19980a3-d265-46e3-be8d-88968dd205fc-kube-api-access-pk2t7" pod "d19980a3-d265-46e3-be8d-88968dd205fc" (UID: "d19980a3-d265-46e3-be8d-88968dd205fc"). InnerVolumeSpecName "kube-api-access-pk2t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:30.510525 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.510488    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:30.515957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.515933    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:30.532188 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.532165    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk2t7\" (UniqueName: \"kubernetes.io/projected/d19980a3-d265-46e3-be8d-88968dd205fc-kube-api-access-pk2t7\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:30.532275 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.532188    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19980a3-d265-46e3-be8d-88968dd205fc-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:30.541560 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.541501    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:30.546833 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:30.546813    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:31.197670 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:31.197643    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth"
Nov 03 20:12:31.197670 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:31.197653    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth" event={"ID":"d19980a3-d265-46e3-be8d-88968dd205fc","Type":"ContainerDied","Data":"a0ad7f6e3803a6873cfc636b31772fba3cce09536713d9e1eaa2182b64116c53"}
Nov 03 20:12:31.198254 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:31.197710    4421 scope.go:122] "RemoveContainer" containerID="f124aa3695f2fa69196ad21e8814480dbd00b7382b8333dad424f91705ceaec1"
Nov 03 20:12:31.200048 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:31.200024    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc" event={"ID":"feb6a504-7873-440d-a013-36654e702307","Type":"ContainerStarted","Data":"b061e3e4a0f02d7919a177b0ecba3e0270825a1ecafaa442df0a21d23bd11cb7"}
Nov 03 20:12:31.200239 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:31.200216    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc"
Nov 03 20:12:31.200367 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:31.200253    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc" event={"ID":"feb6a504-7873-440d-a013-36654e702307","Type":"ContainerStarted","Data":"a033bfe1e3d51daed22dec547fa78b98df9b6d9ae395af6b2536551fa52554a9"}
Nov 03 20:12:31.202036 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:31.201962    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:31.202036 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:31.202008    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:31.202750 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:31.202728    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:31.202857 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:31.202809    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:31.269160 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:31.269091    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm" podUID="f88f1ea1-6c54-4741-bb9c-0676f02b6ff1" containerName="payload-pre-processing" containerID="cri-o://baf1581fc7392b77554607551928601006c08a22a3106eb674efb4cbc928de8b" gracePeriod=25
Nov 03 20:12:31.272132 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:31.272089    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc" podStartSLOduration=2.272070705 podStartE2EDuration="2.272070705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:31.270172018 +0000 UTC m=+681.403300481" watchObservedRunningTime="2026-11-03 20:12:31.272070705 +0000 UTC m=+681.405199168"
Nov 03 20:12:31.363180 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:31.363140    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth"]
Nov 03 20:12:31.369102 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:31.369077    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9dd42e13-6df76474f8-xjlth"]
Nov 03 20:12:31.578160 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:31.578133    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm"
Nov 03 20:12:31.643150 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:31.643119    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f88f1ea1-6c54-4741-bb9c-0676f02b6ff1-kube-api-access-rtmq8\" (UniqueName: \"kubernetes.io/projected/f88f1ea1-6c54-4741-bb9c-0676f02b6ff1-kube-api-access-rtmq8\") pod \"f88f1ea1-6c54-4741-bb9c-0676f02b6ff1\" (UID: \"f88f1ea1-6c54-4741-bb9c-0676f02b6ff1\") "
Nov 03 20:12:31.643329 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:31.643200    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f88f1ea1-6c54-4741-bb9c-0676f02b6ff1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f88f1ea1-6c54-4741-bb9c-0676f02b6ff1-plugins-config-volume\") pod \"f88f1ea1-6c54-4741-bb9c-0676f02b6ff1\" (UID: \"f88f1ea1-6c54-4741-bb9c-0676f02b6ff1\") "
Nov 03 20:12:31.643661 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:31.643630    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f88f1ea1-6c54-4741-bb9c-0676f02b6ff1-plugins-config-volume" pod "f88f1ea1-6c54-4741-bb9c-0676f02b6ff1" (UID: "f88f1ea1-6c54-4741-bb9c-0676f02b6ff1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:31.645432 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:31.645411    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f88f1ea1-6c54-4741-bb9c-0676f02b6ff1-kube-api-access-rtmq8" pod "f88f1ea1-6c54-4741-bb9c-0676f02b6ff1" (UID: "f88f1ea1-6c54-4741-bb9c-0676f02b6ff1"). InnerVolumeSpecName "kube-api-access-rtmq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:31.744831 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:31.744756    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f88f1ea1-6c54-4741-bb9c-0676f02b6ff1-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:31.744831 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:31.744783    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtmq8\" (UniqueName: \"kubernetes.io/projected/f88f1ea1-6c54-4741-bb9c-0676f02b6ff1-kube-api-access-rtmq8\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:31.865833 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:31.865800    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4"]
Nov 03 20:12:31.866090 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:31.866063    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4" podUID="b5e02496-3aae-4a0e-9925-36742914626b" containerName="maas-api" containerID="cri-o://8f0c3b3f1382a689b5abd6cff62c0dd98bd7e8d8aa00833f73935a399195a0fe" gracePeriod=30
Nov 03 20:12:31.955939 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:31.955892    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2f29441c-68679b869d-kd68l"]
Nov 03 20:12:31.964901 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:31.964872    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq"]
Nov 03 20:12:32.017009 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.016324    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="981a2ed3-5af9-45ff-b165-283348d9942e" path="/var/lib/kubelet/pods/981a2ed3-5af9-45ff-b165-283348d9942e/volumes"
Nov 03 20:12:32.018416 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.017458    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d19980a3-d265-46e3-be8d-88968dd205fc" path="/var/lib/kubelet/pods/d19980a3-d265-46e3-be8d-88968dd205fc/volumes"
Nov 03 20:12:32.127526 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.127246    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:12:32.166032 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.164937    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-token\" (UniqueName: \"kubernetes.io/projected/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-token\") pod \"ecbd165b-b0c7-436d-b1ef-6dae96349766\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") "
Nov 03 20:12:32.166032 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.164988    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecbd165b-b0c7-436d-b1ef-6dae96349766-kube-api-access-5t5w7\" (UniqueName: \"kubernetes.io/projected/ecbd165b-b0c7-436d-b1ef-6dae96349766-kube-api-access-5t5w7\") pod \"ecbd165b-b0c7-436d-b1ef-6dae96349766\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") "
Nov 03 20:12:32.166032 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.165097    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-podinfo\") pod \"ecbd165b-b0c7-436d-b1ef-6dae96349766\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") "
Nov 03 20:12:32.166032 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.165135    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecbd165b-b0c7-436d-b1ef-6dae96349766-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecbd165b-b0c7-436d-b1ef-6dae96349766-istiod-ca-cert\") pod \"ecbd165b-b0c7-436d-b1ef-6dae96349766\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") "
Nov 03 20:12:32.166032 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.165172    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-data\") pod \"ecbd165b-b0c7-436d-b1ef-6dae96349766\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") "
Nov 03 20:12:32.166032 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.165217    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-workload-socket\") pod \"ecbd165b-b0c7-436d-b1ef-6dae96349766\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") "
Nov 03 20:12:32.166032 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.165246    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-workload-certs\") pod \"ecbd165b-b0c7-436d-b1ef-6dae96349766\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") "
Nov 03 20:12:32.166032 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.165302    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-credential-socket\") pod \"ecbd165b-b0c7-436d-b1ef-6dae96349766\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") "
Nov 03 20:12:32.166032 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.165351    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-envoy\") pod \"ecbd165b-b0c7-436d-b1ef-6dae96349766\" (UID: \"ecbd165b-b0c7-436d-b1ef-6dae96349766\") "
Nov 03 20:12:32.167416 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.167118    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecbd165b-b0c7-436d-b1ef-6dae96349766-istiod-ca-cert" pod "ecbd165b-b0c7-436d-b1ef-6dae96349766" (UID: "ecbd165b-b0c7-436d-b1ef-6dae96349766"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:32.182164 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.181886    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-workload-socket" pod "ecbd165b-b0c7-436d-b1ef-6dae96349766" (UID: "ecbd165b-b0c7-436d-b1ef-6dae96349766"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:32.182164 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.182029    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-data" pod "ecbd165b-b0c7-436d-b1ef-6dae96349766" (UID: "ecbd165b-b0c7-436d-b1ef-6dae96349766"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:32.182164 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.182093    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-workload-certs" pod "ecbd165b-b0c7-436d-b1ef-6dae96349766" (UID: "ecbd165b-b0c7-436d-b1ef-6dae96349766"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:32.182164 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.182130    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-credential-socket" pod "ecbd165b-b0c7-436d-b1ef-6dae96349766" (UID: "ecbd165b-b0c7-436d-b1ef-6dae96349766"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:32.184712 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.184669    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-podinfo" pod "ecbd165b-b0c7-436d-b1ef-6dae96349766" (UID: "ecbd165b-b0c7-436d-b1ef-6dae96349766"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:12:32.215789 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.215754    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-token" pod "ecbd165b-b0c7-436d-b1ef-6dae96349766" (UID: "ecbd165b-b0c7-436d-b1ef-6dae96349766"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:32.216168 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.215961    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-envoy" pod "ecbd165b-b0c7-436d-b1ef-6dae96349766" (UID: "ecbd165b-b0c7-436d-b1ef-6dae96349766"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:12:32.216168 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.216029    4421 generic.go:356] "Generic (PLEG): container finished" podID="ecbd165b-b0c7-436d-b1ef-6dae96349766" containerID="b0fa49cf5ca7b76a18bfadcfadfa401a0cdd162cbd19ac7cad6840dcb3d13836" exitCode=0
Nov 03 20:12:32.216168 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.216108    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm" event={"ID":"ecbd165b-b0c7-436d-b1ef-6dae96349766","Type":"ContainerDied","Data":"b0fa49cf5ca7b76a18bfadcfadfa401a0cdd162cbd19ac7cad6840dcb3d13836"}
Nov 03 20:12:32.216168 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.216145    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"
Nov 03 20:12:32.216168 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.216166    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm" event={"ID":"ecbd165b-b0c7-436d-b1ef-6dae96349766","Type":"ContainerDied","Data":"545dfa3ccf251be19e08618c07ee7d17017d2010e7fcef6cbcc87c95de2883f9"}
Nov 03 20:12:32.216422 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.216192    4421 scope.go:122] "RemoveContainer" containerID="b0fa49cf5ca7b76a18bfadcfadfa401a0cdd162cbd19ac7cad6840dcb3d13836"
Nov 03 20:12:32.219764 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.219738    4421 generic.go:356] "Generic (PLEG): container finished" podID="b5e02496-3aae-4a0e-9925-36742914626b" containerID="8f0c3b3f1382a689b5abd6cff62c0dd98bd7e8d8aa00833f73935a399195a0fe" exitCode=0
Nov 03 20:12:32.219883 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.219829    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4" event={"ID":"b5e02496-3aae-4a0e-9925-36742914626b","Type":"ContainerDied","Data":"8f0c3b3f1382a689b5abd6cff62c0dd98bd7e8d8aa00833f73935a399195a0fe"}
Nov 03 20:12:32.220120 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.220094    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecbd165b-b0c7-436d-b1ef-6dae96349766-kube-api-access-5t5w7" pod "ecbd165b-b0c7-436d-b1ef-6dae96349766" (UID: "ecbd165b-b0c7-436d-b1ef-6dae96349766"). InnerVolumeSpecName "kube-api-access-5t5w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:32.221222 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.221184    4421 generic.go:356] "Generic (PLEG): container finished" podID="f88f1ea1-6c54-4741-bb9c-0676f02b6ff1" containerID="baf1581fc7392b77554607551928601006c08a22a3106eb674efb4cbc928de8b" exitCode=0
Nov 03 20:12:32.221312 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.221261    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm" event={"ID":"f88f1ea1-6c54-4741-bb9c-0676f02b6ff1","Type":"ContainerDied","Data":"baf1581fc7392b77554607551928601006c08a22a3106eb674efb4cbc928de8b"}
Nov 03 20:12:32.221374 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.221311    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm" event={"ID":"f88f1ea1-6c54-4741-bb9c-0676f02b6ff1","Type":"ContainerDied","Data":"2e1a62b80486927e1a336f42acc9902b4223ca4d0815d9422f583fbf23e222a5"}
Nov 03 20:12:32.221374 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.221273    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm"
Nov 03 20:12:32.222826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.222797    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4"
Nov 03 20:12:32.227121 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.227103    4421 scope.go:122] "RemoveContainer" containerID="fceba77512b24390bbb55cedf987f2684ea797234fc9972a11576f2e046fb96d"
Nov 03 20:12:32.235731 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.235713    4421 scope.go:122] "RemoveContainer" containerID="b0fa49cf5ca7b76a18bfadcfadfa401a0cdd162cbd19ac7cad6840dcb3d13836"
Nov 03 20:12:32.236021 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:12:32.236000    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0fa49cf5ca7b76a18bfadcfadfa401a0cdd162cbd19ac7cad6840dcb3d13836\": container with ID starting with b0fa49cf5ca7b76a18bfadcfadfa401a0cdd162cbd19ac7cad6840dcb3d13836 not found: ID does not exist" containerID="b0fa49cf5ca7b76a18bfadcfadfa401a0cdd162cbd19ac7cad6840dcb3d13836"
Nov 03 20:12:32.236080 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.236041    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0fa49cf5ca7b76a18bfadcfadfa401a0cdd162cbd19ac7cad6840dcb3d13836"} err="failed to get container status \"b0fa49cf5ca7b76a18bfadcfadfa401a0cdd162cbd19ac7cad6840dcb3d13836\": rpc error: code = NotFound desc = could not find container \"b0fa49cf5ca7b76a18bfadcfadfa401a0cdd162cbd19ac7cad6840dcb3d13836\": container with ID starting with b0fa49cf5ca7b76a18bfadcfadfa401a0cdd162cbd19ac7cad6840dcb3d13836 not found: ID does not exist"
Nov 03 20:12:32.236080 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.236057    4421 scope.go:122] "RemoveContainer" containerID="fceba77512b24390bbb55cedf987f2684ea797234fc9972a11576f2e046fb96d"
Nov 03 20:12:32.236298 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:12:32.236284    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fceba77512b24390bbb55cedf987f2684ea797234fc9972a11576f2e046fb96d\": container with ID starting with fceba77512b24390bbb55cedf987f2684ea797234fc9972a11576f2e046fb96d not found: ID does not exist" containerID="fceba77512b24390bbb55cedf987f2684ea797234fc9972a11576f2e046fb96d"
Nov 03 20:12:32.236363 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.236301    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fceba77512b24390bbb55cedf987f2684ea797234fc9972a11576f2e046fb96d"} err="failed to get container status \"fceba77512b24390bbb55cedf987f2684ea797234fc9972a11576f2e046fb96d\": rpc error: code = NotFound desc = could not find container \"fceba77512b24390bbb55cedf987f2684ea797234fc9972a11576f2e046fb96d\": container with ID starting with fceba77512b24390bbb55cedf987f2684ea797234fc9972a11576f2e046fb96d not found: ID does not exist"
Nov 03 20:12:32.236363 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.236311    4421 scope.go:122] "RemoveContainer" containerID="baf1581fc7392b77554607551928601006c08a22a3106eb674efb4cbc928de8b"
Nov 03 20:12:32.246664 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.246641    4421 scope.go:122] "RemoveContainer" containerID="baf1581fc7392b77554607551928601006c08a22a3106eb674efb4cbc928de8b"
Nov 03 20:12:32.248701 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:12:32.247066    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baf1581fc7392b77554607551928601006c08a22a3106eb674efb4cbc928de8b\": container with ID starting with baf1581fc7392b77554607551928601006c08a22a3106eb674efb4cbc928de8b not found: ID does not exist" containerID="baf1581fc7392b77554607551928601006c08a22a3106eb674efb4cbc928de8b"
Nov 03 20:12:32.248701 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.247099    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baf1581fc7392b77554607551928601006c08a22a3106eb674efb4cbc928de8b"} err="failed to get container status \"baf1581fc7392b77554607551928601006c08a22a3106eb674efb4cbc928de8b\": rpc error: code = NotFound desc = could not find container \"baf1581fc7392b77554607551928601006c08a22a3106eb674efb4cbc928de8b\": container with ID starting with baf1581fc7392b77554607551928601006c08a22a3106eb674efb4cbc928de8b not found: ID does not exist"
Nov 03 20:12:32.265870 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.265837    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm"]
Nov 03 20:12:32.266070 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.266052    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5e02496-3aae-4a0e-9925-36742914626b-kube-api-access-pq2n6\" (UniqueName: \"kubernetes.io/projected/b5e02496-3aae-4a0e-9925-36742914626b-kube-api-access-pq2n6\") pod \"b5e02496-3aae-4a0e-9925-36742914626b\" (UID: \"b5e02496-3aae-4a0e-9925-36742914626b\") "
Nov 03 20:12:32.266112 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.266095    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5e02496-3aae-4a0e-9925-36742914626b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5e02496-3aae-4a0e-9925-36742914626b-maas-api-tls\") pod \"b5e02496-3aae-4a0e-9925-36742914626b\" (UID: \"b5e02496-3aae-4a0e-9925-36742914626b\") "
Nov 03 20:12:32.266178 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.266165    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5e02496-3aae-4a0e-9925-36742914626b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5e02496-3aae-4a0e-9925-36742914626b-metrics-certs\") pod \"b5e02496-3aae-4a0e-9925-36742914626b\" (UID: \"b5e02496-3aae-4a0e-9925-36742914626b\") "
Nov 03 20:12:32.266257 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.266241    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5e02496-3aae-4a0e-9925-36742914626b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5e02496-3aae-4a0e-9925-36742914626b-openshift-service-ca\") pod \"b5e02496-3aae-4a0e-9925-36742914626b\" (UID: \"b5e02496-3aae-4a0e-9925-36742914626b\") "
Nov 03 20:12:32.266727 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.266699    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5e02496-3aae-4a0e-9925-36742914626b-openshift-service-ca" pod "b5e02496-3aae-4a0e-9925-36742914626b" (UID: "b5e02496-3aae-4a0e-9925-36742914626b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:32.266975 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.266956    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5e02496-3aae-4a0e-9925-36742914626b-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:32.267072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.266981    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:32.267072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.266995    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecbd165b-b0c7-436d-b1ef-6dae96349766-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:32.267072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.267009    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:32.267072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.267024    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:32.267072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.267038    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:32.267072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.267052    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:32.267072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.267064    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:32.267437 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.267078    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecbd165b-b0c7-436d-b1ef-6dae96349766-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:32.267437 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.267092    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t5w7\" (UniqueName: \"kubernetes.io/projected/ecbd165b-b0c7-436d-b1ef-6dae96349766-kube-api-access-5t5w7\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:32.269362 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.269321    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5e02496-3aae-4a0e-9925-36742914626b-maas-api-tls" pod "b5e02496-3aae-4a0e-9925-36742914626b" (UID: "b5e02496-3aae-4a0e-9925-36742914626b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:12:32.271351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.271314    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5e02496-3aae-4a0e-9925-36742914626b-kube-api-access-pq2n6" pod "b5e02496-3aae-4a0e-9925-36742914626b" (UID: "b5e02496-3aae-4a0e-9925-36742914626b"). InnerVolumeSpecName "kube-api-access-pq2n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:32.273852 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.273825    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4bd752fd-56bfc9f7dc-r8sxm"]
Nov 03 20:12:32.281203 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.281176    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5e02496-3aae-4a0e-9925-36742914626b-metrics-certs" pod "b5e02496-3aae-4a0e-9925-36742914626b" (UID: "b5e02496-3aae-4a0e-9925-36742914626b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:12:32.368131 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.368099    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5e02496-3aae-4a0e-9925-36742914626b-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:32.368131 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.368130    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq2n6\" (UniqueName: \"kubernetes.io/projected/b5e02496-3aae-4a0e-9925-36742914626b-kube-api-access-pq2n6\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:32.368351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.368141    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5e02496-3aae-4a0e-9925-36742914626b-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:32.546017 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.545985    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"]
Nov 03 20:12:32.552880 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.552848    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9dd42e13-openshift-default-5897cdd6cc-w28mm"]
Nov 03 20:12:32.603324 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.603293    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p"]
Nov 03 20:12:32.603738 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.603725    4421 cpu_manager.go:414] "containerMap: removing container" podUID="981a2ed3-5af9-45ff-b165-283348d9942e" containerName="payload-pre-processing"
Nov 03 20:12:32.603787 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.603740    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="981a2ed3-5af9-45ff-b165-283348d9942e" containerName="payload-pre-processing"
Nov 03 20:12:32.603787 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.603752    4421 cpu_manager.go:414] "containerMap: removing container" podUID="ecbd165b-b0c7-436d-b1ef-6dae96349766" containerName="istio-proxy"
Nov 03 20:12:32.603787 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.603758    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecbd165b-b0c7-436d-b1ef-6dae96349766" containerName="istio-proxy"
Nov 03 20:12:32.603787 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.603774    4421 cpu_manager.go:414] "containerMap: removing container" podUID="d19980a3-d265-46e3-be8d-88968dd205fc" containerName="payload-processing"
Nov 03 20:12:32.603787 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.603779    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d19980a3-d265-46e3-be8d-88968dd205fc" containerName="payload-processing"
Nov 03 20:12:32.603931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.603790    4421 cpu_manager.go:414] "containerMap: removing container" podUID="ecbd165b-b0c7-436d-b1ef-6dae96349766" containerName="istio-proxy"
Nov 03 20:12:32.603931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.603795    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecbd165b-b0c7-436d-b1ef-6dae96349766" containerName="istio-proxy"
Nov 03 20:12:32.603931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.603802    4421 cpu_manager.go:414] "containerMap: removing container" podUID="b5e02496-3aae-4a0e-9925-36742914626b" containerName="maas-api"
Nov 03 20:12:32.603931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.603806    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5e02496-3aae-4a0e-9925-36742914626b" containerName="maas-api"
Nov 03 20:12:32.603931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.603821    4421 cpu_manager.go:414] "containerMap: removing container" podUID="f88f1ea1-6c54-4741-bb9c-0676f02b6ff1" containerName="payload-pre-processing"
Nov 03 20:12:32.603931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.603825    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f88f1ea1-6c54-4741-bb9c-0676f02b6ff1" containerName="payload-pre-processing"
Nov 03 20:12:32.603931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.603881    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f88f1ea1-6c54-4741-bb9c-0676f02b6ff1" containerName="payload-pre-processing"
Nov 03 20:12:32.603931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.603888    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="981a2ed3-5af9-45ff-b165-283348d9942e" containerName="payload-pre-processing"
Nov 03 20:12:32.603931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.603894    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecbd165b-b0c7-436d-b1ef-6dae96349766" containerName="istio-proxy"
Nov 03 20:12:32.603931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.603901    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d19980a3-d265-46e3-be8d-88968dd205fc" containerName="payload-processing"
Nov 03 20:12:32.603931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.603907    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5e02496-3aae-4a0e-9925-36742914626b" containerName="maas-api"
Nov 03 20:12:32.608472 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.608451    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p"
Nov 03 20:12:32.618661 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.618638    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-466f83\""
Nov 03 20:12:32.622726 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.621541    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p"]
Nov 03 20:12:32.670687 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.670656    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1d072eb-7506-40d4-ad37-99b01543157b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p\" (UID: \"c1d072eb-7506-40d4-ad37-99b01543157b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p"
Nov 03 20:12:32.670811 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.670792    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n96m8\" (UniqueName: \"kubernetes.io/projected/c1d072eb-7506-40d4-ad37-99b01543157b-kube-api-access-n96m8\") pod \"payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p\" (UID: \"c1d072eb-7506-40d4-ad37-99b01543157b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p"
Nov 03 20:12:32.685687 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.685664    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j"]
Nov 03 20:12:32.686182 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.686167    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecbd165b-b0c7-436d-b1ef-6dae96349766" containerName="istio-proxy"
Nov 03 20:12:32.689271 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.689257    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j"
Nov 03 20:12:32.698673 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.698653    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-466f83-dockercfg-6xlr6\""
Nov 03 20:12:32.701840 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.701817    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j"]
Nov 03 20:12:32.772014 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.771983    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5d225a-004c-45c2-944b-987796507d35-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j\" (UID: \"2f5d225a-004c-45c2-944b-987796507d35\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j"
Nov 03 20:12:32.772165 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.772038    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v78l4\" (UniqueName: \"kubernetes.io/projected/2f5d225a-004c-45c2-944b-987796507d35-kube-api-access-v78l4\") pod \"payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j\" (UID: \"2f5d225a-004c-45c2-944b-987796507d35\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j"
Nov 03 20:12:32.772165 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.772140    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n96m8\" (UniqueName: \"kubernetes.io/projected/c1d072eb-7506-40d4-ad37-99b01543157b-kube-api-access-n96m8\") pod \"payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p\" (UID: \"c1d072eb-7506-40d4-ad37-99b01543157b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p"
Nov 03 20:12:32.772254 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.772210    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1d072eb-7506-40d4-ad37-99b01543157b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p\" (UID: \"c1d072eb-7506-40d4-ad37-99b01543157b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p"
Nov 03 20:12:32.772898 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.772881    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1d072eb-7506-40d4-ad37-99b01543157b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p\" (UID: \"c1d072eb-7506-40d4-ad37-99b01543157b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p"
Nov 03 20:12:32.824937 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.824863    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n96m8\" (UniqueName: \"kubernetes.io/projected/c1d072eb-7506-40d4-ad37-99b01543157b-kube-api-access-n96m8\") pod \"payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p\" (UID: \"c1d072eb-7506-40d4-ad37-99b01543157b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p"
Nov 03 20:12:32.873731 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.873687    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5d225a-004c-45c2-944b-987796507d35-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j\" (UID: \"2f5d225a-004c-45c2-944b-987796507d35\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j"
Nov 03 20:12:32.873930 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.873788    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v78l4\" (UniqueName: \"kubernetes.io/projected/2f5d225a-004c-45c2-944b-987796507d35-kube-api-access-v78l4\") pod \"payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j\" (UID: \"2f5d225a-004c-45c2-944b-987796507d35\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j"
Nov 03 20:12:32.874754 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.874727    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5d225a-004c-45c2-944b-987796507d35-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j\" (UID: \"2f5d225a-004c-45c2-944b-987796507d35\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j"
Nov 03 20:12:32.894089 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.894057    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v78l4\" (UniqueName: \"kubernetes.io/projected/2f5d225a-004c-45c2-944b-987796507d35-kube-api-access-v78l4\") pod \"payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j\" (UID: \"2f5d225a-004c-45c2-944b-987796507d35\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j"
Nov 03 20:12:32.925808 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:32.925778    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p"
Nov 03 20:12:33.004681 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:33.004308    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j"
Nov 03 20:12:33.211071 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:33.211043    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p"]
Nov 03 20:12:33.213233 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:12:33.213204    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d072eb_7506_40d4_ad37_99b01543157b.slice/crio-d218371806974215f1c8231efbf731c5f9a78bf81db766bbadcf2d6784bfe800 WatchSource:0}: Error finding container d218371806974215f1c8231efbf731c5f9a78bf81db766bbadcf2d6784bfe800: Status 404 returned error can't find the container with id d218371806974215f1c8231efbf731c5f9a78bf81db766bbadcf2d6784bfe800
Nov 03 20:12:33.227085 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:33.227067    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4"
Nov 03 20:12:33.227463 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:33.227085    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4" event={"ID":"b5e02496-3aae-4a0e-9925-36742914626b","Type":"ContainerDied","Data":"f86819f306aaf682a02efcc0a3cca87ee5af5efabea8a573f442215625836364"}
Nov 03 20:12:33.227463 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:33.227144    4421 scope.go:122] "RemoveContainer" containerID="8f0c3b3f1382a689b5abd6cff62c0dd98bd7e8d8aa00833f73935a399195a0fe"
Nov 03 20:12:33.234691 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:33.234635    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p" event={"ID":"c1d072eb-7506-40d4-ad37-99b01543157b","Type":"ContainerStarted","Data":"d218371806974215f1c8231efbf731c5f9a78bf81db766bbadcf2d6784bfe800"}
Nov 03 20:12:33.276707 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:33.276685    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4"]
Nov 03 20:12:33.284368 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:33.284310    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2f29441c-6b7b7dd965-bjpq4"]
Nov 03 20:12:33.308186 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:12:33.308158    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f5d225a_004c_45c2_944b_987796507d35.slice/crio-7e3b05cea975107549774c600a94a8015ec3ebe28de33fecbfb22728c6f2e749 WatchSource:0}: Error finding container 7e3b05cea975107549774c600a94a8015ec3ebe28de33fecbfb22728c6f2e749: Status 404 returned error can't find the container with id 7e3b05cea975107549774c600a94a8015ec3ebe28de33fecbfb22728c6f2e749
Nov 03 20:12:33.309156 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:33.309128    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j"]
Nov 03 20:12:34.015098 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.015052    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5e02496-3aae-4a0e-9925-36742914626b" path="/var/lib/kubelet/pods/b5e02496-3aae-4a0e-9925-36742914626b/volumes"
Nov 03 20:12:34.015526 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.015511    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecbd165b-b0c7-436d-b1ef-6dae96349766" path="/var/lib/kubelet/pods/ecbd165b-b0c7-436d-b1ef-6dae96349766/volumes"
Nov 03 20:12:34.015927 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.015913    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f88f1ea1-6c54-4741-bb9c-0676f02b6ff1" path="/var/lib/kubelet/pods/f88f1ea1-6c54-4741-bb9c-0676f02b6ff1/volumes"
Nov 03 20:12:34.243084 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.243050    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p" event={"ID":"c1d072eb-7506-40d4-ad37-99b01543157b","Type":"ContainerStarted","Data":"9ab8b35faa70f7c5a3375091c8f80e209f202f2aafc2bc0e0937b0cbbbc15ff2"}
Nov 03 20:12:34.243637 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.243614    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p"
Nov 03 20:12:34.244880 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.244857    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j" event={"ID":"2f5d225a-004c-45c2-944b-987796507d35","Type":"ContainerStarted","Data":"eeaab1516e5bd83d097d73cdb6ea781a4add20e3dad00228943eaa179a3aec20"}
Nov 03 20:12:34.244981 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.244897    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j" event={"ID":"2f5d225a-004c-45c2-944b-987796507d35","Type":"ContainerStarted","Data":"7e3b05cea975107549774c600a94a8015ec3ebe28de33fecbfb22728c6f2e749"}
Nov 03 20:12:34.244981 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.244941    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j"
Nov 03 20:12:34.277259 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.277190    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"]
Nov 03 20:12:34.293120 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.293093    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn"]
Nov 03 20:12:34.293256 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.293205    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.297178 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.297154    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn"
Nov 03 20:12:34.311224 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.311201    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b13a14-openshift-default-dockercfg-r6zzp\""
Nov 03 20:12:34.314670 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.314613    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p" podStartSLOduration=2.314578955 podStartE2EDuration="2.314578955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:34.309228642 +0000 UTC m=+684.442357105" watchObservedRunningTime="2026-11-03 20:12:34.314578955 +0000 UTC m=+684.447707417"
Nov 03 20:12:34.320007 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.319684    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"]
Nov 03 20:12:34.333130 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.328846    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn"]
Nov 03 20:12:34.382811 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.382764    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j" podStartSLOduration=2.382750124 podStartE2EDuration="2.382750124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:34.382050635 +0000 UTC m=+684.515179096" watchObservedRunningTime="2026-11-03 20:12:34.382750124 +0000 UTC m=+684.515878586"
Nov 03 20:12:34.393428 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.393399    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.393547 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.393435    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b270c5c6-0444-4410-a70d-37edb799adce-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.393547 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.393465    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs6l9\" (UniqueName: \"kubernetes.io/projected/b270c5c6-0444-4410-a70d-37edb799adce-kube-api-access-xs6l9\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.393683 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.393586    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.393683 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.393646    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.393783 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.393744    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b270c5c6-0444-4410-a70d-37edb799adce-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.393825 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.393804    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2mzc\" (UniqueName: \"kubernetes.io/projected/3f2eb82c-c1c9-4895-97bb-0ca733b765f3-kube-api-access-l2mzc\") pod \"payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn\" (UID: \"3f2eb82c-c1c9-4895-97bb-0ca733b765f3\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn"
Nov 03 20:12:34.393858 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.393827    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.393858 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.393843    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b270c5c6-0444-4410-a70d-37edb799adce-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.393957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.393879    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.393957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.393919    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f2eb82c-c1c9-4895-97bb-0ca733b765f3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn\" (UID: \"3f2eb82c-c1c9-4895-97bb-0ca733b765f3\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn"
Nov 03 20:12:34.414179 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.414153    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f3631f\""
Nov 03 20:12:34.495020 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.494980    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs6l9\" (UniqueName: \"kubernetes.io/projected/b270c5c6-0444-4410-a70d-37edb799adce-kube-api-access-xs6l9\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.495201 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.495051    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.495201 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.495091    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.495201 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.495171    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b270c5c6-0444-4410-a70d-37edb799adce-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.495369 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.495220    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2mzc\" (UniqueName: \"kubernetes.io/projected/3f2eb82c-c1c9-4895-97bb-0ca733b765f3-kube-api-access-l2mzc\") pod \"payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn\" (UID: \"3f2eb82c-c1c9-4895-97bb-0ca733b765f3\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn"
Nov 03 20:12:34.495369 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.495253    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.495369 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.495280    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b270c5c6-0444-4410-a70d-37edb799adce-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.495369 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.495336    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.495556 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.495381    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f2eb82c-c1c9-4895-97bb-0ca733b765f3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn\" (UID: \"3f2eb82c-c1c9-4895-97bb-0ca733b765f3\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn"
Nov 03 20:12:34.495556 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.495449    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.495556 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.495473    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b270c5c6-0444-4410-a70d-37edb799adce-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.495556 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.495501    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.495556 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.495542    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.495828 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.495796    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.496138 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.496117    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.496687 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.496552    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f2eb82c-c1c9-4895-97bb-0ca733b765f3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn\" (UID: \"3f2eb82c-c1c9-4895-97bb-0ca733b765f3\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn"
Nov 03 20:12:34.496780 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.496685    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b270c5c6-0444-4410-a70d-37edb799adce-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.498006 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.497978    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b270c5c6-0444-4410-a70d-37edb799adce-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.498150 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.498134    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.507459 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.507439    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b270c5c6-0444-4410-a70d-37edb799adce-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.507710 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.507688    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2mzc\" (UniqueName: \"kubernetes.io/projected/3f2eb82c-c1c9-4895-97bb-0ca733b765f3-kube-api-access-l2mzc\") pod \"payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn\" (UID: \"3f2eb82c-c1c9-4895-97bb-0ca733b765f3\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn"
Nov 03 20:12:34.508502 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.508482    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs6l9\" (UniqueName: \"kubernetes.io/projected/b270c5c6-0444-4410-a70d-37edb799adce-kube-api-access-xs6l9\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.608484 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.608395    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:34.619286 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.619256    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn"
Nov 03 20:12:34.919561 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.919531    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"]
Nov 03 20:12:34.921225 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:12:34.921199    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb270c5c6_0444_4410_a70d_37edb799adce.slice/crio-0c9722064644a7ef9ec7c09a4599d0391704ad7a921790ac7a0ff6963ada8f61 WatchSource:0}: Error finding container 0c9722064644a7ef9ec7c09a4599d0391704ad7a921790ac7a0ff6963ada8f61: Status 404 returned error can't find the container with id 0c9722064644a7ef9ec7c09a4599d0391704ad7a921790ac7a0ff6963ada8f61
Nov 03 20:12:34.923571 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.923540    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:34.923660 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.923624    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:34.923697 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.923660    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:34.936757 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:34.936737    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn"]
Nov 03 20:12:34.938857 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:12:34.938834    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f2eb82c_c1c9_4895_97bb_0ca733b765f3.slice/crio-9663ec291b15e7b6a09a29777502d42e4e47eee93e9187887cade15371cf6fea WatchSource:0}: Error finding container 9663ec291b15e7b6a09a29777502d42e4e47eee93e9187887cade15371cf6fea: Status 404 returned error can't find the container with id 9663ec291b15e7b6a09a29777502d42e4e47eee93e9187887cade15371cf6fea
Nov 03 20:12:35.253087 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:35.253056    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47" event={"ID":"b270c5c6-0444-4410-a70d-37edb799adce","Type":"ContainerStarted","Data":"2884e4fab020ea09b5e6f5d307306ff6e6778c87b614512d886f9c67b009a517"}
Nov 03 20:12:35.253564 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:35.253144    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47" event={"ID":"b270c5c6-0444-4410-a70d-37edb799adce","Type":"ContainerStarted","Data":"0c9722064644a7ef9ec7c09a4599d0391704ad7a921790ac7a0ff6963ada8f61"}
Nov 03 20:12:35.254903 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:35.254878    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn" event={"ID":"3f2eb82c-c1c9-4895-97bb-0ca733b765f3","Type":"ContainerStarted","Data":"544c28eeb8b30ffe14c7a6b83852a70e70ac213fa239300a9dcd498734a8bbfb"}
Nov 03 20:12:35.255000 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:35.254922    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn" event={"ID":"3f2eb82c-c1c9-4895-97bb-0ca733b765f3","Type":"ContainerStarted","Data":"9663ec291b15e7b6a09a29777502d42e4e47eee93e9187887cade15371cf6fea"}
Nov 03 20:12:35.255110 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:35.255092    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn"
Nov 03 20:12:35.279396 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:35.279351    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47" podStartSLOduration=1.27933802 podStartE2EDuration="1.27933802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:35.276945224 +0000 UTC m=+685.410073705" watchObservedRunningTime="2026-11-03 20:12:35.27933802 +0000 UTC m=+685.412466482"
Nov 03 20:12:35.301082 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:35.301034    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn" podStartSLOduration=1.301022661 podStartE2EDuration="1.301022661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:35.299849413 +0000 UTC m=+685.432977896" watchObservedRunningTime="2026-11-03 20:12:35.301022661 +0000 UTC m=+685.434151122"
Nov 03 20:12:35.609498 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:35.609416    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:35.615797 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:35.615762    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:36.260649 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:36.260621    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:36.261544 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:36.261525    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:12:36.422462 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:36.422416    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9"]
Nov 03 20:12:36.429620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:36.429579    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9"
Nov 03 20:12:36.433959 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:36.433934    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-085566-serving-cert\""
Nov 03 20:12:36.445091 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:36.443712    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9"]
Nov 03 20:12:36.559266 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:36.559188    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afdf3167-09b7-498c-8674-31b63af5622c-metrics-certs\") pod \"maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9\" (UID: \"afdf3167-09b7-498c-8674-31b63af5622c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9"
Nov 03 20:12:36.559266 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:36.559243    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n2hk\" (UniqueName: \"kubernetes.io/projected/afdf3167-09b7-498c-8674-31b63af5622c-kube-api-access-2n2hk\") pod \"maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9\" (UID: \"afdf3167-09b7-498c-8674-31b63af5622c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9"
Nov 03 20:12:36.559445 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:36.559350    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afdf3167-09b7-498c-8674-31b63af5622c-maas-api-tls\") pod \"maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9\" (UID: \"afdf3167-09b7-498c-8674-31b63af5622c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9"
Nov 03 20:12:36.559501 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:36.559483    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afdf3167-09b7-498c-8674-31b63af5622c-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9\" (UID: \"afdf3167-09b7-498c-8674-31b63af5622c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9"
Nov 03 20:12:36.660644 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:36.660566    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afdf3167-09b7-498c-8674-31b63af5622c-maas-api-tls\") pod \"maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9\" (UID: \"afdf3167-09b7-498c-8674-31b63af5622c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9"
Nov 03 20:12:36.660795 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:36.660699    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afdf3167-09b7-498c-8674-31b63af5622c-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9\" (UID: \"afdf3167-09b7-498c-8674-31b63af5622c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9"
Nov 03 20:12:36.660795 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:36.660734    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afdf3167-09b7-498c-8674-31b63af5622c-metrics-certs\") pod \"maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9\" (UID: \"afdf3167-09b7-498c-8674-31b63af5622c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9"
Nov 03 20:12:36.660795 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:36.660771    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n2hk\" (UniqueName: \"kubernetes.io/projected/afdf3167-09b7-498c-8674-31b63af5622c-kube-api-access-2n2hk\") pod \"maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9\" (UID: \"afdf3167-09b7-498c-8674-31b63af5622c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9"
Nov 03 20:12:36.661427 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:36.661403    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afdf3167-09b7-498c-8674-31b63af5622c-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9\" (UID: \"afdf3167-09b7-498c-8674-31b63af5622c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9"
Nov 03 20:12:36.663387 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:36.663365    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afdf3167-09b7-498c-8674-31b63af5622c-metrics-certs\") pod \"maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9\" (UID: \"afdf3167-09b7-498c-8674-31b63af5622c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9"
Nov 03 20:12:36.663825 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:36.663792    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afdf3167-09b7-498c-8674-31b63af5622c-maas-api-tls\") pod \"maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9\" (UID: \"afdf3167-09b7-498c-8674-31b63af5622c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9"
Nov 03 20:12:36.673576 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:36.673554    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n2hk\" (UniqueName: \"kubernetes.io/projected/afdf3167-09b7-498c-8674-31b63af5622c-kube-api-access-2n2hk\") pod \"maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9\" (UID: \"afdf3167-09b7-498c-8674-31b63af5622c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9"
Nov 03 20:12:36.754892 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:36.754860    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9"
Nov 03 20:12:36.956878 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:36.956810    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-2f29441c-68679b869d-kd68l" podUID="e3fdb849-094e-42a1-a807-d663a6356515" containerName="payload-processing" containerID="cri-o://cd5058d996b51caecf539b5411019fc54d9a835b7ba0614a38eb38d4c4661fa7" gracePeriod=25
Nov 03 20:12:36.965569 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:36.965520    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq" podUID="4c519650-430a-4012-ab7e-3eeb88567c36" containerName="payload-pre-processing" containerID="cri-o://8062c73a8d58cad2ac9bb0996379406d7352d2be9d7de28af990ab85bb92859a" gracePeriod=25
Nov 03 20:12:37.038490 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:37.038468    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9"]
Nov 03 20:12:37.053736 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:12:37.053695    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafdf3167_09b7_498c_8674_31b63af5622c.slice/crio-412ebb9594ea13592ccc14835dc493463397ba0749b6bebd9ec2d8b3a5fc8b04 WatchSource:0}: Error finding container 412ebb9594ea13592ccc14835dc493463397ba0749b6bebd9ec2d8b3a5fc8b04: Status 404 returned error can't find the container with id 412ebb9594ea13592ccc14835dc493463397ba0749b6bebd9ec2d8b3a5fc8b04
Nov 03 20:12:37.259723 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:37.259697    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2f29441c-68679b869d-kd68l"
Nov 03 20:12:37.278207 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:37.278159    4421 generic.go:356] "Generic (PLEG): container finished" podID="4c519650-430a-4012-ab7e-3eeb88567c36" containerID="8062c73a8d58cad2ac9bb0996379406d7352d2be9d7de28af990ab85bb92859a" exitCode=0
Nov 03 20:12:37.278637 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:37.278331    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq" event={"ID":"4c519650-430a-4012-ab7e-3eeb88567c36","Type":"ContainerDied","Data":"8062c73a8d58cad2ac9bb0996379406d7352d2be9d7de28af990ab85bb92859a"}
Nov 03 20:12:37.282122 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:37.281751    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9" event={"ID":"afdf3167-09b7-498c-8674-31b63af5622c","Type":"ContainerStarted","Data":"412ebb9594ea13592ccc14835dc493463397ba0749b6bebd9ec2d8b3a5fc8b04"}
Nov 03 20:12:37.284137 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:37.283172    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2f29441c-68679b869d-kd68l" event={"ID":"e3fdb849-094e-42a1-a807-d663a6356515","Type":"ContainerDied","Data":"cd5058d996b51caecf539b5411019fc54d9a835b7ba0614a38eb38d4c4661fa7"}
Nov 03 20:12:37.284137 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:37.283229    4421 scope.go:122] "RemoveContainer" containerID="cd5058d996b51caecf539b5411019fc54d9a835b7ba0614a38eb38d4c4661fa7"
Nov 03 20:12:37.284137 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:37.283278    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2f29441c-68679b869d-kd68l"
Nov 03 20:12:37.284137 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:37.283169    4421 generic.go:356] "Generic (PLEG): container finished" podID="e3fdb849-094e-42a1-a807-d663a6356515" containerID="cd5058d996b51caecf539b5411019fc54d9a835b7ba0614a38eb38d4c4661fa7" exitCode=0
Nov 03 20:12:37.284137 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:37.283394    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2f29441c-68679b869d-kd68l" event={"ID":"e3fdb849-094e-42a1-a807-d663a6356515","Type":"ContainerDied","Data":"772f12dc47daca2b19a6d91f3011b14bf6585347cda7475f7cc122dead084e60"}
Nov 03 20:12:37.297874 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:37.294417    4421 scope.go:122] "RemoveContainer" containerID="cd5058d996b51caecf539b5411019fc54d9a835b7ba0614a38eb38d4c4661fa7"
Nov 03 20:12:37.297874 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:12:37.294769    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd5058d996b51caecf539b5411019fc54d9a835b7ba0614a38eb38d4c4661fa7\": container with ID starting with cd5058d996b51caecf539b5411019fc54d9a835b7ba0614a38eb38d4c4661fa7 not found: ID does not exist" containerID="cd5058d996b51caecf539b5411019fc54d9a835b7ba0614a38eb38d4c4661fa7"
Nov 03 20:12:37.297874 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:37.294805    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd5058d996b51caecf539b5411019fc54d9a835b7ba0614a38eb38d4c4661fa7"} err="failed to get container status \"cd5058d996b51caecf539b5411019fc54d9a835b7ba0614a38eb38d4c4661fa7\": rpc error: code = NotFound desc = could not find container \"cd5058d996b51caecf539b5411019fc54d9a835b7ba0614a38eb38d4c4661fa7\": container with ID starting with cd5058d996b51caecf539b5411019fc54d9a835b7ba0614a38eb38d4c4661fa7 not found: ID does not exist"
Nov 03 20:12:37.324526 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:37.324457    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq"
Nov 03 20:12:37.400047 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:37.400015    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3fdb849-094e-42a1-a807-d663a6356515-kube-api-access-vjdq5\" (UniqueName: \"kubernetes.io/projected/e3fdb849-094e-42a1-a807-d663a6356515-kube-api-access-vjdq5\") pod \"e3fdb849-094e-42a1-a807-d663a6356515\" (UID: \"e3fdb849-094e-42a1-a807-d663a6356515\") "
Nov 03 20:12:37.400221 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:37.400199    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3fdb849-094e-42a1-a807-d663a6356515-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3fdb849-094e-42a1-a807-d663a6356515-plugins-config-volume\") pod \"e3fdb849-094e-42a1-a807-d663a6356515\" (UID: \"e3fdb849-094e-42a1-a807-d663a6356515\") "
Nov 03 20:12:37.400663 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:37.400630    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3fdb849-094e-42a1-a807-d663a6356515-plugins-config-volume" pod "e3fdb849-094e-42a1-a807-d663a6356515" (UID: "e3fdb849-094e-42a1-a807-d663a6356515"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:37.402385 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:37.402363    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3fdb849-094e-42a1-a807-d663a6356515-kube-api-access-vjdq5" pod "e3fdb849-094e-42a1-a807-d663a6356515" (UID: "e3fdb849-094e-42a1-a807-d663a6356515"). InnerVolumeSpecName "kube-api-access-vjdq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:37.502054 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:37.502019    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c519650-430a-4012-ab7e-3eeb88567c36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c519650-430a-4012-ab7e-3eeb88567c36-plugins-config-volume\") pod \"4c519650-430a-4012-ab7e-3eeb88567c36\" (UID: \"4c519650-430a-4012-ab7e-3eeb88567c36\") "
Nov 03 20:12:37.502491 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:37.502458    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c519650-430a-4012-ab7e-3eeb88567c36-plugins-config-volume" pod "4c519650-430a-4012-ab7e-3eeb88567c36" (UID: "4c519650-430a-4012-ab7e-3eeb88567c36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:37.502491 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:37.502457    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c519650-430a-4012-ab7e-3eeb88567c36-kube-api-access-f56fq\" (UniqueName: \"kubernetes.io/projected/4c519650-430a-4012-ab7e-3eeb88567c36-kube-api-access-f56fq\") pod \"4c519650-430a-4012-ab7e-3eeb88567c36\" (UID: \"4c519650-430a-4012-ab7e-3eeb88567c36\") "
Nov 03 20:12:37.502906 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:37.502886    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjdq5\" (UniqueName: \"kubernetes.io/projected/e3fdb849-094e-42a1-a807-d663a6356515-kube-api-access-vjdq5\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:37.502990 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:37.502909    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c519650-430a-4012-ab7e-3eeb88567c36-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:37.502990 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:37.502922    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3fdb849-094e-42a1-a807-d663a6356515-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:37.504727 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:37.504705    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c519650-430a-4012-ab7e-3eeb88567c36-kube-api-access-f56fq" pod "4c519650-430a-4012-ab7e-3eeb88567c36" (UID: "4c519650-430a-4012-ab7e-3eeb88567c36"). InnerVolumeSpecName "kube-api-access-f56fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:37.604011 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:37.603989    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f56fq\" (UniqueName: \"kubernetes.io/projected/4c519650-430a-4012-ab7e-3eeb88567c36-kube-api-access-f56fq\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:37.672638 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:37.670680    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2f29441c-68679b869d-kd68l"]
Nov 03 20:12:37.689179 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:37.689138    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2f29441c-68679b869d-kd68l"]
Nov 03 20:12:38.014049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:38.014016    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3fdb849-094e-42a1-a807-d663a6356515" path="/var/lib/kubelet/pods/e3fdb849-094e-42a1-a807-d663a6356515/volumes"
Nov 03 20:12:38.291284 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:38.291200    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq"
Nov 03 20:12:38.291284 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:38.291240    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq" event={"ID":"4c519650-430a-4012-ab7e-3eeb88567c36","Type":"ContainerDied","Data":"ceef58f7fc6ba0617dbb996d7c29b47b5c19ce5ec940e345932c63ad34f3529a"}
Nov 03 20:12:38.291843 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:38.291313    4421 scope.go:122] "RemoveContainer" containerID="8062c73a8d58cad2ac9bb0996379406d7352d2be9d7de28af990ab85bb92859a"
Nov 03 20:12:38.293073 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:38.293048    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9" event={"ID":"afdf3167-09b7-498c-8674-31b63af5622c","Type":"ContainerStarted","Data":"fa633f0aedf2ecedf54908a8fe247937f538067ef00b0d7d2205fc9ceba8bc6d"}
Nov 03 20:12:38.293176 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:38.293106    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9"
Nov 03 20:12:38.309134 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:38.309113    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq"]
Nov 03 20:12:38.315441 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:38.315248    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2f29441c-5c594d8f6c-btlkq"]
Nov 03 20:12:39.175489 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:39.175426    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9" podStartSLOduration=2.705518422 podStartE2EDuration="3.175405966s" totalImagesPullingTime="469.887544ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:38.342881136 +0000 UTC m=+688.476009614" watchObservedRunningTime="2026-11-03 20:12:39.175405966 +0000 UTC m=+689.308534429"
Nov 03 20:12:39.176192 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:39.176174    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww"]
Nov 03 20:12:39.176673 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:39.176659    4421 cpu_manager.go:414] "containerMap: removing container" podUID="4c519650-430a-4012-ab7e-3eeb88567c36" containerName="payload-pre-processing"
Nov 03 20:12:39.176734 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:39.176674    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c519650-430a-4012-ab7e-3eeb88567c36" containerName="payload-pre-processing"
Nov 03 20:12:39.176734 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:39.176707    4421 cpu_manager.go:414] "containerMap: removing container" podUID="e3fdb849-094e-42a1-a807-d663a6356515" containerName="payload-processing"
Nov 03 20:12:39.176734 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:39.176712    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3fdb849-094e-42a1-a807-d663a6356515" containerName="payload-processing"
Nov 03 20:12:39.176854 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:39.176776    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3fdb849-094e-42a1-a807-d663a6356515" containerName="payload-processing"
Nov 03 20:12:39.176854 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:39.176786    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c519650-430a-4012-ab7e-3eeb88567c36" containerName="payload-pre-processing"
Nov 03 20:12:39.181234 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:39.181212    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww"
Nov 03 20:12:39.191234 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:39.191211    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-b13a14\""
Nov 03 20:12:39.191530 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:39.191509    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-b13a14-dockercfg-qlnfs\""
Nov 03 20:12:39.194590 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:39.194400    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww"]
Nov 03 20:12:39.334881 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:39.334834    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzrd8\" (UniqueName: \"kubernetes.io/projected/5107af0d-00d7-415e-be8e-e9ce4d55f8c0-kube-api-access-kzrd8\") pod \"payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww\" (UID: \"5107af0d-00d7-415e-be8e-e9ce4d55f8c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww"
Nov 03 20:12:39.355652 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:39.347964    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5107af0d-00d7-415e-be8e-e9ce4d55f8c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww\" (UID: \"5107af0d-00d7-415e-be8e-e9ce4d55f8c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww"
Nov 03 20:12:39.449609 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:39.449570    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5107af0d-00d7-415e-be8e-e9ce4d55f8c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww\" (UID: \"5107af0d-00d7-415e-be8e-e9ce4d55f8c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww"
Nov 03 20:12:39.449800 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:39.449759    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzrd8\" (UniqueName: \"kubernetes.io/projected/5107af0d-00d7-415e-be8e-e9ce4d55f8c0-kube-api-access-kzrd8\") pod \"payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww\" (UID: \"5107af0d-00d7-415e-be8e-e9ce4d55f8c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww"
Nov 03 20:12:39.450345 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:39.450322    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5107af0d-00d7-415e-be8e-e9ce4d55f8c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww\" (UID: \"5107af0d-00d7-415e-be8e-e9ce4d55f8c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww"
Nov 03 20:12:39.462673 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:39.462644    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzrd8\" (UniqueName: \"kubernetes.io/projected/5107af0d-00d7-415e-be8e-e9ce4d55f8c0-kube-api-access-kzrd8\") pod \"payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww\" (UID: \"5107af0d-00d7-415e-be8e-e9ce4d55f8c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww"
Nov 03 20:12:39.501958 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:39.501932    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww"
Nov 03 20:12:39.780511 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:39.780484    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww"]
Nov 03 20:12:39.782326 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:12:39.782298    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5107af0d_00d7_415e_be8e_e9ce4d55f8c0.slice/crio-52be822881c58ec844b7e189f8f8dfb53de1d4c889c967650929ee3bcc3c2bf3 WatchSource:0}: Error finding container 52be822881c58ec844b7e189f8f8dfb53de1d4c889c967650929ee3bcc3c2bf3: Status 404 returned error can't find the container with id 52be822881c58ec844b7e189f8f8dfb53de1d4c889c967650929ee3bcc3c2bf3
Nov 03 20:12:40.016466 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:40.016420    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c519650-430a-4012-ab7e-3eeb88567c36" path="/var/lib/kubelet/pods/4c519650-430a-4012-ab7e-3eeb88567c36/volumes"
Nov 03 20:12:40.394639 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:40.394548    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww" event={"ID":"5107af0d-00d7-415e-be8e-e9ce4d55f8c0","Type":"ContainerStarted","Data":"d97ee714e89844c09b2c11978f09d50562d9c4e5b58a809deb0a41176784122d"}
Nov 03 20:12:40.394639 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:40.394640    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww" event={"ID":"5107af0d-00d7-415e-be8e-e9ce4d55f8c0","Type":"ContainerStarted","Data":"52be822881c58ec844b7e189f8f8dfb53de1d4c889c967650929ee3bcc3c2bf3"}
Nov 03 20:12:40.395222 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:40.394940    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww"
Nov 03 20:12:40.415856 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:40.415814    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww" podStartSLOduration=1.415800741 podStartE2EDuration="1.415800741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:40.414388407 +0000 UTC m=+690.547516872" watchObservedRunningTime="2026-11-03 20:12:40.415800741 +0000 UTC m=+690.548929203"
Nov 03 20:12:41.667623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:41.666647    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4"]
Nov 03 20:12:41.677965 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:41.672262    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4"
Nov 03 20:12:41.699621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:41.681130    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-b13a14-serving-cert\""
Nov 03 20:12:41.704628 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:41.703707    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4"]
Nov 03 20:12:41.807231 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:41.807186    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/070c9447-94dc-4af0-90a1-ad328f85bd84-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4\" (UID: \"070c9447-94dc-4af0-90a1-ad328f85bd84\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4"
Nov 03 20:12:41.807591 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:41.807524    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpwqb\" (UniqueName: \"kubernetes.io/projected/070c9447-94dc-4af0-90a1-ad328f85bd84-kube-api-access-mpwqb\") pod \"maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4\" (UID: \"070c9447-94dc-4af0-90a1-ad328f85bd84\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4"
Nov 03 20:12:41.807929 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:41.807900    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/070c9447-94dc-4af0-90a1-ad328f85bd84-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4\" (UID: \"070c9447-94dc-4af0-90a1-ad328f85bd84\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4"
Nov 03 20:12:41.808137 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:41.808110    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/070c9447-94dc-4af0-90a1-ad328f85bd84-metrics-certs\") pod \"maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4\" (UID: \"070c9447-94dc-4af0-90a1-ad328f85bd84\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4"
Nov 03 20:12:41.912469 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:41.912426    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpwqb\" (UniqueName: \"kubernetes.io/projected/070c9447-94dc-4af0-90a1-ad328f85bd84-kube-api-access-mpwqb\") pod \"maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4\" (UID: \"070c9447-94dc-4af0-90a1-ad328f85bd84\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4"
Nov 03 20:12:41.912696 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:41.912501    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/070c9447-94dc-4af0-90a1-ad328f85bd84-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4\" (UID: \"070c9447-94dc-4af0-90a1-ad328f85bd84\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4"
Nov 03 20:12:41.912696 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:41.912573    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/070c9447-94dc-4af0-90a1-ad328f85bd84-metrics-certs\") pod \"maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4\" (UID: \"070c9447-94dc-4af0-90a1-ad328f85bd84\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4"
Nov 03 20:12:41.912696 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:41.912668    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/070c9447-94dc-4af0-90a1-ad328f85bd84-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4\" (UID: \"070c9447-94dc-4af0-90a1-ad328f85bd84\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4"
Nov 03 20:12:41.913805 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:41.913716    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/070c9447-94dc-4af0-90a1-ad328f85bd84-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4\" (UID: \"070c9447-94dc-4af0-90a1-ad328f85bd84\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4"
Nov 03 20:12:41.915416 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:41.915367    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/070c9447-94dc-4af0-90a1-ad328f85bd84-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4\" (UID: \"070c9447-94dc-4af0-90a1-ad328f85bd84\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4"
Nov 03 20:12:41.916199 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:41.916174    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/070c9447-94dc-4af0-90a1-ad328f85bd84-metrics-certs\") pod \"maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4\" (UID: \"070c9447-94dc-4af0-90a1-ad328f85bd84\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4"
Nov 03 20:12:41.929815 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:41.929753    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpwqb\" (UniqueName: \"kubernetes.io/projected/070c9447-94dc-4af0-90a1-ad328f85bd84-kube-api-access-mpwqb\") pod \"maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4\" (UID: \"070c9447-94dc-4af0-90a1-ad328f85bd84\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4"
Nov 03 20:12:42.023237 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:42.023210    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4"
Nov 03 20:12:42.225542 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:42.225458    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc"
Nov 03 20:12:42.397626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:42.395436    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4"]
Nov 03 20:12:42.409287 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:12:42.408350    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070c9447_94dc_4af0_90a1_ad328f85bd84.slice/crio-0b52ff34df6e3c391a3c9eb6eda421eda9068abf7ad09f8596c2b413e293958d WatchSource:0}: Error finding container 0b52ff34df6e3c391a3c9eb6eda421eda9068abf7ad09f8596c2b413e293958d: Status 404 returned error can't find the container with id 0b52ff34df6e3c391a3c9eb6eda421eda9068abf7ad09f8596c2b413e293958d
Nov 03 20:12:43.419754 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:43.419661    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4" event={"ID":"070c9447-94dc-4af0-90a1-ad328f85bd84","Type":"ContainerStarted","Data":"3c9c7cb8c51c1f3f643938ca775e6cfff7b104b236027c262c5ba658cad15bcc"}
Nov 03 20:12:43.419754 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:43.419721    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4" event={"ID":"070c9447-94dc-4af0-90a1-ad328f85bd84","Type":"ContainerStarted","Data":"0b52ff34df6e3c391a3c9eb6eda421eda9068abf7ad09f8596c2b413e293958d"}
Nov 03 20:12:43.419754 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:43.419745    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4"
Nov 03 20:12:43.440395 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:43.440351    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4" podStartSLOduration=1.842199613 podStartE2EDuration="2.440307774s" totalImagesPullingTime="598.108161ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:43.439151307 +0000 UTC m=+693.572279794" watchObservedRunningTime="2026-11-03 20:12:43.440307774 +0000 UTC m=+693.573436286"
Nov 03 20:12:44.395629 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:44.393774    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9"
Nov 03 20:12:45.277624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:45.274766    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j"
Nov 03 20:12:46.282877 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:46.282084    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p"
Nov 03 20:12:46.284152 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:46.283511    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn"
Nov 03 20:12:47.245801 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:47.245725    4421 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:12:47.245989 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:47.245816    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4" podUID="830b16e0-459d-444c-b7da-c8d6d8576d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:12:47.248824 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:47.248699    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:12:47.248824 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:47.248753    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw" podUID="317e6b68-68f8-4220-a3ef-2f0999a19ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:12:49.432062 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:49.432029    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4"
Nov 03 20:12:49.631697 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:12:49.631662    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:12:50.486195 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:50.486158    4421 generic.go:356] "Generic (PLEG): container finished" podID="317e6b68-68f8-4220-a3ef-2f0999a19ad9" containerID="759d180f655772ea7fe565676cfd10e5575a2ae5a37aa2bb9454a3c57baee927" exitCode=0
Nov 03 20:12:50.486729 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:50.486226    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw" event={"ID":"317e6b68-68f8-4220-a3ef-2f0999a19ad9","Type":"ContainerDied","Data":"759d180f655772ea7fe565676cfd10e5575a2ae5a37aa2bb9454a3c57baee927"}
Nov 03 20:12:50.486729 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:50.486675    4421 scope.go:122] "RemoveContainer" containerID="759d180f655772ea7fe565676cfd10e5575a2ae5a37aa2bb9454a3c57baee927"
Nov 03 20:12:50.487977 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:50.487958    4421 generic.go:356] "Generic (PLEG): container finished" podID="830b16e0-459d-444c-b7da-c8d6d8576d4f" containerID="296173dacd6b5ac0ad08d8f5cb4c6356ebbde4afc2a09ebf492e25eb42971f2b" exitCode=0
Nov 03 20:12:50.488073 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:50.488039    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4" event={"ID":"830b16e0-459d-444c-b7da-c8d6d8576d4f","Type":"ContainerDied","Data":"296173dacd6b5ac0ad08d8f5cb4c6356ebbde4afc2a09ebf492e25eb42971f2b"}
Nov 03 20:12:50.488140 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:50.488051    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:50.488208 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:50.488191    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:50.488267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:50.488230    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:50.488440 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:50.488425    4421 scope.go:122] "RemoveContainer" containerID="296173dacd6b5ac0ad08d8f5cb4c6356ebbde4afc2a09ebf492e25eb42971f2b"
Nov 03 20:12:50.489683 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:50.489658    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:50.489796 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:50.489717    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:50.489796 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:50.489791    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:51.417624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:51.417232    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww"
Nov 03 20:12:51.509637 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:51.506849    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw" event={"ID":"317e6b68-68f8-4220-a3ef-2f0999a19ad9","Type":"ContainerStarted","Data":"2b14fce9a07d8e0822f9cb6964c3462892594276505a91b7dc0a9ff28e28e167"}
Nov 03 20:12:51.514867 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:51.514831    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4" event={"ID":"830b16e0-459d-444c-b7da-c8d6d8576d4f","Type":"ContainerStarted","Data":"57d92c1a1e5f97709e501e78b53a1cda2b96502f2d7197edce56424ab1b29889"}
Nov 03 20:12:51.548624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:51.544969    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:51.548624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:51.545086    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 20:12:51.548624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:51.545137    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw" podUID="317e6b68-68f8-4220-a3ef-2f0999a19ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 20:12:52.512623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:52.511786    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:52.512623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:52.512500    4421 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 20:12:52.513193 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:52.512623    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4" podUID="830b16e0-459d-444c-b7da-c8d6d8576d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 20:12:52.582623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:52.551050    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 20:12:52.582623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:52.551134    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw" podUID="317e6b68-68f8-4220-a3ef-2f0999a19ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 20:12:53.511625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:53.511008    4421 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 20:12:53.511625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:53.511105    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4" podUID="830b16e0-459d-444c-b7da-c8d6d8576d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 20:12:53.550387 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:53.549971    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 20:12:53.550387 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:53.550050    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw" podUID="317e6b68-68f8-4220-a3ef-2f0999a19ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 20:12:54.510487 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:54.510441    4421 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 20:12:54.510685 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:54.510511    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4" podUID="830b16e0-459d-444c-b7da-c8d6d8576d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 20:12:54.560680 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:54.560026    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 20:12:54.560680 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:54.560086    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw" podUID="317e6b68-68f8-4220-a3ef-2f0999a19ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 20:12:55.511061 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:55.511011    4421 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 20:12:55.511251 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:55.511068    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4" podUID="830b16e0-459d-444c-b7da-c8d6d8576d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 20:12:55.545187 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:55.545139    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 20:12:55.545482 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:55.545427    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw" podUID="317e6b68-68f8-4220-a3ef-2f0999a19ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 20:12:56.510086 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:56.510042    4421 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 20:12:56.510478 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:56.510128    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4" podUID="830b16e0-459d-444c-b7da-c8d6d8576d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 20:12:56.560627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:56.557072    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 20:12:56.560627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:56.557143    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw" podUID="317e6b68-68f8-4220-a3ef-2f0999a19ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 20:12:57.515779 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:57.515750    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:57.516178 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:57.516135    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:57.516832 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:57.516813    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:12:57.547266 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:57.547234    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:57.548551 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:57.548529    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:57.549622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:57.549573    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:12:58.453969 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.453935    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"]
Nov 03 20:12:58.456578 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.456552    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:12:58.467290 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.467144    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-085566\"/\"simulator-w2-085566-kserve-self-signed-certs\""
Nov 03 20:12:58.467290 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.467144    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-085566\"/\"default-dockercfg-5n6wm\""
Nov 03 20:12:58.467290 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.467200    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-085566\"/\"kube-root-ca.crt\""
Nov 03 20:12:58.467290 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.467260    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-085566\"/\"openshift-service-ca.crt\""
Nov 03 20:12:58.469706 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.469685    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"]
Nov 03 20:12:58.498467 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.498431    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-dshm\") pod \"simulator-w2-085566-kserve-5959946fbc-56t6t\" (UID: \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\") " pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:12:58.498653 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.498507    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-tmp-dir\") pod \"simulator-w2-085566-kserve-5959946fbc-56t6t\" (UID: \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\") " pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:12:58.498653 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.498633    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd24563c-4cba-4ad1-9f2b-0f62b199634f-tls-certs\") pod \"simulator-w2-085566-kserve-5959946fbc-56t6t\" (UID: \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\") " pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:12:58.498808 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.498786    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-home\") pod \"simulator-w2-085566-kserve-5959946fbc-56t6t\" (UID: \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\") " pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:12:58.498949 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.498874    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-model-cache\") pod \"simulator-w2-085566-kserve-5959946fbc-56t6t\" (UID: \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\") " pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:12:58.500170 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.500147    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbrj7\" (UniqueName: \"kubernetes.io/projected/bd24563c-4cba-4ad1-9f2b-0f62b199634f-kube-api-access-kbrj7\") pod \"simulator-w2-085566-kserve-5959946fbc-56t6t\" (UID: \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\") " pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:12:58.601004 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.600926    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-home\") pod \"simulator-w2-085566-kserve-5959946fbc-56t6t\" (UID: \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\") " pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:12:58.601004 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.600994    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-model-cache\") pod \"simulator-w2-085566-kserve-5959946fbc-56t6t\" (UID: \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\") " pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:12:58.601513 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.601068    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbrj7\" (UniqueName: \"kubernetes.io/projected/bd24563c-4cba-4ad1-9f2b-0f62b199634f-kube-api-access-kbrj7\") pod \"simulator-w2-085566-kserve-5959946fbc-56t6t\" (UID: \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\") " pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:12:58.601513 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.601116    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-dshm\") pod \"simulator-w2-085566-kserve-5959946fbc-56t6t\" (UID: \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\") " pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:12:58.601513 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.601170    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-tmp-dir\") pod \"simulator-w2-085566-kserve-5959946fbc-56t6t\" (UID: \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\") " pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:12:58.601513 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.601211    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd24563c-4cba-4ad1-9f2b-0f62b199634f-tls-certs\") pod \"simulator-w2-085566-kserve-5959946fbc-56t6t\" (UID: \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\") " pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:12:58.601513 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.601345    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-home\") pod \"simulator-w2-085566-kserve-5959946fbc-56t6t\" (UID: \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\") " pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:12:58.601513 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.601408    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-model-cache\") pod \"simulator-w2-085566-kserve-5959946fbc-56t6t\" (UID: \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\") " pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:12:58.601855 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.601556    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-tmp-dir\") pod \"simulator-w2-085566-kserve-5959946fbc-56t6t\" (UID: \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\") " pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:12:58.603527 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.603499    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-dshm\") pod \"simulator-w2-085566-kserve-5959946fbc-56t6t\" (UID: \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\") " pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:12:58.603813 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.603793    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd24563c-4cba-4ad1-9f2b-0f62b199634f-tls-certs\") pod \"simulator-w2-085566-kserve-5959946fbc-56t6t\" (UID: \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\") " pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:12:58.614548 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.614526    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbrj7\" (UniqueName: \"kubernetes.io/projected/bd24563c-4cba-4ad1-9f2b-0f62b199634f-kube-api-access-kbrj7\") pod \"simulator-w2-085566-kserve-5959946fbc-56t6t\" (UID: \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\") " pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:12:58.627542 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.627519    4421 generic.go:356] "Generic (PLEG): container finished" podID="b270c5c6-0444-4410-a70d-37edb799adce" containerID="2884e4fab020ea09b5e6f5d307306ff6e6778c87b614512d886f9c67b009a517" exitCode=0
Nov 03 20:12:58.627688 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.627624    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47" event={"ID":"b270c5c6-0444-4410-a70d-37edb799adce","Type":"ContainerDied","Data":"2884e4fab020ea09b5e6f5d307306ff6e6778c87b614512d886f9c67b009a517"}
Nov 03 20:12:58.628391 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.628373    4421 scope.go:122] "RemoveContainer" containerID="2884e4fab020ea09b5e6f5d307306ff6e6778c87b614512d886f9c67b009a517"
Nov 03 20:12:58.629526 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.629498    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:58.629591 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.629563    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:58.629655 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.629610    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:58.789276 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:58.788996    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:12:59.074880 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.074849    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"]
Nov 03 20:12:59.077437 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:12:59.077403    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd24563c_4cba_4ad1_9f2b_0f62b199634f.slice/crio-a2200a736b42d3f20b1d2a897b971923aaeb3db6133210c64deb0c04fdf5376f WatchSource:0}: Error finding container a2200a736b42d3f20b1d2a897b971923aaeb3db6133210c64deb0c04fdf5376f: Status 404 returned error can't find the container with id a2200a736b42d3f20b1d2a897b971923aaeb3db6133210c64deb0c04fdf5376f
Nov 03 20:12:59.558881 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.541072    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"]
Nov 03 20:12:59.576620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.576435    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"]
Nov 03 20:12:59.576620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.576557    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:12:59.580675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.580650    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f3631f\"/\"default-dockercfg-wp86k\""
Nov 03 20:12:59.581911 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.581885    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f3631f\"/\"simulator-w0-f3631f-kserve-self-signed-certs\""
Nov 03 20:12:59.582263 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.582233    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f3631f\"/\"kube-root-ca.crt\""
Nov 03 20:12:59.582367 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.582315    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f3631f\"/\"openshift-service-ca.crt\""
Nov 03 20:12:59.615773 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.615743    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-home\") pod \"simulator-w0-f3631f-kserve-b4fb8c546-6f2qn\" (UID: \"4abd92cb-973c-4d9d-9a7e-003109eef888\") " pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:12:59.616167 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.615813    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-dshm\") pod \"simulator-w0-f3631f-kserve-b4fb8c546-6f2qn\" (UID: \"4abd92cb-973c-4d9d-9a7e-003109eef888\") " pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:12:59.616167 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.615887    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-model-cache\") pod \"simulator-w0-f3631f-kserve-b4fb8c546-6f2qn\" (UID: \"4abd92cb-973c-4d9d-9a7e-003109eef888\") " pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:12:59.616167 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.615926    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9t4x\" (UniqueName: \"kubernetes.io/projected/4abd92cb-973c-4d9d-9a7e-003109eef888-kube-api-access-p9t4x\") pod \"simulator-w0-f3631f-kserve-b4fb8c546-6f2qn\" (UID: \"4abd92cb-973c-4d9d-9a7e-003109eef888\") " pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:12:59.616167 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.615947    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4abd92cb-973c-4d9d-9a7e-003109eef888-tls-certs\") pod \"simulator-w0-f3631f-kserve-b4fb8c546-6f2qn\" (UID: \"4abd92cb-973c-4d9d-9a7e-003109eef888\") " pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:12:59.616167 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.616028    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-tmp-dir\") pod \"simulator-w0-f3631f-kserve-b4fb8c546-6f2qn\" (UID: \"4abd92cb-973c-4d9d-9a7e-003109eef888\") " pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:12:59.634370 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.634334    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" event={"ID":"bd24563c-4cba-4ad1-9f2b-0f62b199634f","Type":"ContainerStarted","Data":"01de387c14daeb12e629246cc7a52c4ecd0b7ad67eb272e186137532d3a37fb0"}
Nov 03 20:12:59.634553 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.634385    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" event={"ID":"bd24563c-4cba-4ad1-9f2b-0f62b199634f","Type":"ContainerStarted","Data":"a2200a736b42d3f20b1d2a897b971923aaeb3db6133210c64deb0c04fdf5376f"}
Nov 03 20:12:59.636230 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.636206    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47" event={"ID":"b270c5c6-0444-4410-a70d-37edb799adce","Type":"ContainerStarted","Data":"8cd02ec096fa460a4056a45596bddfb301f6dd5867c22a355a4c8d44d4c0052d"}
Nov 03 20:12:59.667572 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.667504    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podStartSLOduration=1.6674820700000001 podStartE2EDuration="1.66748207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:59.665911278 +0000 UTC m=+709.799039741" watchObservedRunningTime="2026-11-03 20:12:59.66748207 +0000 UTC m=+709.800610533"
Nov 03 20:12:59.716792 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.716757    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-home\") pod \"simulator-w0-f3631f-kserve-b4fb8c546-6f2qn\" (UID: \"4abd92cb-973c-4d9d-9a7e-003109eef888\") " pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:12:59.716971 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.716944    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-dshm\") pod \"simulator-w0-f3631f-kserve-b4fb8c546-6f2qn\" (UID: \"4abd92cb-973c-4d9d-9a7e-003109eef888\") " pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:12:59.717045 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.716988    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-model-cache\") pod \"simulator-w0-f3631f-kserve-b4fb8c546-6f2qn\" (UID: \"4abd92cb-973c-4d9d-9a7e-003109eef888\") " pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:12:59.717045 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.717023    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9t4x\" (UniqueName: \"kubernetes.io/projected/4abd92cb-973c-4d9d-9a7e-003109eef888-kube-api-access-p9t4x\") pod \"simulator-w0-f3631f-kserve-b4fb8c546-6f2qn\" (UID: \"4abd92cb-973c-4d9d-9a7e-003109eef888\") " pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:12:59.717150 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.717059    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4abd92cb-973c-4d9d-9a7e-003109eef888-tls-certs\") pod \"simulator-w0-f3631f-kserve-b4fb8c546-6f2qn\" (UID: \"4abd92cb-973c-4d9d-9a7e-003109eef888\") " pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:12:59.717150 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.717134    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-tmp-dir\") pod \"simulator-w0-f3631f-kserve-b4fb8c546-6f2qn\" (UID: \"4abd92cb-973c-4d9d-9a7e-003109eef888\") " pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:12:59.717288 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.717265    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-home\") pod \"simulator-w0-f3631f-kserve-b4fb8c546-6f2qn\" (UID: \"4abd92cb-973c-4d9d-9a7e-003109eef888\") " pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:12:59.717353 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.717305    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-model-cache\") pod \"simulator-w0-f3631f-kserve-b4fb8c546-6f2qn\" (UID: \"4abd92cb-973c-4d9d-9a7e-003109eef888\") " pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:12:59.717540 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.717510    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-tmp-dir\") pod \"simulator-w0-f3631f-kserve-b4fb8c546-6f2qn\" (UID: \"4abd92cb-973c-4d9d-9a7e-003109eef888\") " pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:12:59.719418 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.719394    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-dshm\") pod \"simulator-w0-f3631f-kserve-b4fb8c546-6f2qn\" (UID: \"4abd92cb-973c-4d9d-9a7e-003109eef888\") " pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:12:59.719672 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.719659    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4abd92cb-973c-4d9d-9a7e-003109eef888-tls-certs\") pod \"simulator-w0-f3631f-kserve-b4fb8c546-6f2qn\" (UID: \"4abd92cb-973c-4d9d-9a7e-003109eef888\") " pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:12:59.728987 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.728969    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9t4x\" (UniqueName: \"kubernetes.io/projected/4abd92cb-973c-4d9d-9a7e-003109eef888-kube-api-access-p9t4x\") pod \"simulator-w0-f3631f-kserve-b4fb8c546-6f2qn\" (UID: \"4abd92cb-973c-4d9d-9a7e-003109eef888\") " pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:12:59.796759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.796682    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:12:59.890968 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:12:59.890883    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:13:00.190245 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:00.190214    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"]
Nov 03 20:13:00.191490 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:13:00.191464    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4abd92cb_973c_4d9d_9a7e_003109eef888.slice/crio-7b5d3d5073761450aced7cd42952a1e1993f8d346f88d2cee8e7b4eaf3e1f955 WatchSource:0}: Error finding container 7b5d3d5073761450aced7cd42952a1e1993f8d346f88d2cee8e7b4eaf3e1f955: Status 404 returned error can't find the container with id 7b5d3d5073761450aced7cd42952a1e1993f8d346f88d2cee8e7b4eaf3e1f955
Nov 03 20:13:00.609460 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:00.609377    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:13:00.609695 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:00.609555    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:13:00.609695 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:00.609638    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47" podUID="b270c5c6-0444-4410-a70d-37edb799adce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 20:13:00.676731 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:00.676514    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" event={"ID":"4abd92cb-973c-4d9d-9a7e-003109eef888","Type":"ContainerStarted","Data":"8753e602801d6ae0203d12dea13adf632b561a41059e7807a0c29b27d5ed8041"}
Nov 03 20:13:00.677253 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:00.676748    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" event={"ID":"4abd92cb-973c-4d9d-9a7e-003109eef888","Type":"ContainerStarted","Data":"7b5d3d5073761450aced7cd42952a1e1993f8d346f88d2cee8e7b4eaf3e1f955"}
Nov 03 20:13:00.697896 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:00.697850    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podStartSLOduration=1.6978388930000001 podStartE2EDuration="1.697838893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:00.697709102 +0000 UTC m=+710.830837567" watchObservedRunningTime="2026-11-03 20:13:00.697838893 +0000 UTC m=+710.830967355"
Nov 03 20:13:00.895796 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:00.895707    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:13:01.609690 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:01.609655    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:13:01.609872 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:01.609711    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47" podUID="b270c5c6-0444-4410-a70d-37edb799adce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 20:13:02.612238 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:02.612193    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:13:02.612899 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:02.612837    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47" podUID="b270c5c6-0444-4410-a70d-37edb799adce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 20:13:03.609177 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:03.609142    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:13:03.609361 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:03.609192    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47" podUID="b270c5c6-0444-4410-a70d-37edb799adce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 20:13:04.609802 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:04.609765    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 03 20:13:04.610240 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:04.609821    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47" podUID="b270c5c6-0444-4410-a70d-37edb799adce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 03 20:13:05.613421 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:05.613391    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:13:05.613945 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:05.613828    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:13:05.614531 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:05.614513    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:13:08.790487 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:08.790459    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:13:08.809718 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:08.809695    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:13:09.785692 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:09.785662    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:13:09.891142 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:09.891112    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:13:09.903742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:09.903719    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:13:10.802469 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:10.802442    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:13:19.627270 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:13:19.627225    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:13:22.022229 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:22.022190    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"]
Nov 03 20:13:22.027314 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:22.027289    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:13:22.035511 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:22.035456    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f3631f\"/\"premium-w0-f3631f-kserve-self-signed-certs\""
Nov 03 20:13:22.039852 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:22.039827    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"]
Nov 03 20:13:22.054747 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:22.054723    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-model-cache\") pod \"premium-w0-f3631f-kserve-6749dbcb5f-hsfpf\" (UID: \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\") " pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:13:22.054890 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:22.054797    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-dshm\") pod \"premium-w0-f3631f-kserve-6749dbcb5f-hsfpf\" (UID: \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\") " pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:13:22.055047 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:22.055028    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-home\") pod \"premium-w0-f3631f-kserve-6749dbcb5f-hsfpf\" (UID: \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\") " pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:13:22.055105 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:22.055080    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-tls-certs\") pod \"premium-w0-f3631f-kserve-6749dbcb5f-hsfpf\" (UID: \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\") " pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:13:22.055145 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:22.055111    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpdw5\" (UniqueName: \"kubernetes.io/projected/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-kube-api-access-kpdw5\") pod \"premium-w0-f3631f-kserve-6749dbcb5f-hsfpf\" (UID: \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\") " pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:13:22.055211 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:22.055196    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-tmp-dir\") pod \"premium-w0-f3631f-kserve-6749dbcb5f-hsfpf\" (UID: \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\") " pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:13:22.184378 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:22.184336    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-model-cache\") pod \"premium-w0-f3631f-kserve-6749dbcb5f-hsfpf\" (UID: \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\") " pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:13:22.185387 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:22.185362    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-dshm\") pod \"premium-w0-f3631f-kserve-6749dbcb5f-hsfpf\" (UID: \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\") " pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:13:22.186014 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:22.185994    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-home\") pod \"premium-w0-f3631f-kserve-6749dbcb5f-hsfpf\" (UID: \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\") " pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:13:22.186481 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:22.186463    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-tls-certs\") pod \"premium-w0-f3631f-kserve-6749dbcb5f-hsfpf\" (UID: \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\") " pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:13:22.186990 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:22.186968    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpdw5\" (UniqueName: \"kubernetes.io/projected/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-kube-api-access-kpdw5\") pod \"premium-w0-f3631f-kserve-6749dbcb5f-hsfpf\" (UID: \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\") " pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:13:22.187584 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:22.187559    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-tmp-dir\") pod \"premium-w0-f3631f-kserve-6749dbcb5f-hsfpf\" (UID: \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\") " pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:13:22.188100 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:22.185227    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-model-cache\") pod \"premium-w0-f3631f-kserve-6749dbcb5f-hsfpf\" (UID: \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\") " pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:13:22.188234 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:22.186396    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-home\") pod \"premium-w0-f3631f-kserve-6749dbcb5f-hsfpf\" (UID: \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\") " pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:13:22.188345 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:22.187952    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-tmp-dir\") pod \"premium-w0-f3631f-kserve-6749dbcb5f-hsfpf\" (UID: \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\") " pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:13:22.206346 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:22.206313    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-tls-certs\") pod \"premium-w0-f3631f-kserve-6749dbcb5f-hsfpf\" (UID: \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\") " pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:13:22.208307 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:22.208274    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-dshm\") pod \"premium-w0-f3631f-kserve-6749dbcb5f-hsfpf\" (UID: \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\") " pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:13:22.218298 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:22.218126    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpdw5\" (UniqueName: \"kubernetes.io/projected/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-kube-api-access-kpdw5\") pod \"premium-w0-f3631f-kserve-6749dbcb5f-hsfpf\" (UID: \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\") " pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:13:22.343312 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:22.343221    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:13:22.618498 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:22.618468    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"]
Nov 03 20:13:22.620006 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:13:22.619982    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea1d6b64_3d8a_4b5a_b37f_c2b110cddb08.slice/crio-535beb0d922cb590f39bd0bf7319140b0c83a0a9e557437021c03694787e3b85 WatchSource:0}: Error finding container 535beb0d922cb590f39bd0bf7319140b0c83a0a9e557437021c03694787e3b85: Status 404 returned error can't find the container with id 535beb0d922cb590f39bd0bf7319140b0c83a0a9e557437021c03694787e3b85
Nov 03 20:13:22.877316 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:22.877219    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" event={"ID":"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08","Type":"ContainerStarted","Data":"e8d7b451765fe5297901715a171e45acaa202a41f4d9a03a603b8ec3a87ef3a8"}
Nov 03 20:13:22.877316 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:22.877264    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" event={"ID":"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08","Type":"ContainerStarted","Data":"535beb0d922cb590f39bd0bf7319140b0c83a0a9e557437021c03694787e3b85"}
Nov 03 20:13:22.898871 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:22.898822    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podStartSLOduration=0.898808408 podStartE2EDuration="898.808408ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:22.8977512 +0000 UTC m=+733.030879667" watchObservedRunningTime="2026-11-03 20:13:22.898808408 +0000 UTC m=+733.031936899"
Nov 03 20:13:23.344017 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:23.343991    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:13:32.344024 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:32.343994    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:13:32.357293 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:32.357269    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:13:32.951327 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:32.951294    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:13:49.643961 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:13:49.643857    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:13:58.210323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.210282    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"]
Nov 03 20:13:58.214425 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.214401    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.218643 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.218622    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-466f83-openshift-default-dockercfg-6pzmx\""
Nov 03 20:13:58.249178 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.249145    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"]
Nov 03 20:13:58.315233 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.315189    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c80431-a853-4012-b239-ea4659c70bf9-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.322573 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.322373    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfrjx\" (UniqueName: \"kubernetes.io/projected/34c80431-a853-4012-b239-ea4659c70bf9-kube-api-access-hfrjx\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.333754 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.333712    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.334053 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.334033    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.334314 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.334297    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.334502 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.334488    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c80431-a853-4012-b239-ea4659c70bf9-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.346408 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.335007    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.346408 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.346144    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c80431-a853-4012-b239-ea4659c70bf9-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.346408 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.346184    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.449889 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.449853    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.450135 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.449400    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.450307 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.450286    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c80431-a853-4012-b239-ea4659c70bf9-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.451143 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.451120    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.451272 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.451164    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c80431-a853-4012-b239-ea4659c70bf9-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.451272 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.451067    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c80431-a853-4012-b239-ea4659c70bf9-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.451272 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.451199    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.451272 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.451264    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c80431-a853-4012-b239-ea4659c70bf9-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.451500 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.451378    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfrjx\" (UniqueName: \"kubernetes.io/projected/34c80431-a853-4012-b239-ea4659c70bf9-kube-api-access-hfrjx\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.451500 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.451406    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.451500 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.451435    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.451500 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.451481    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.452081 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.452054    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.452419 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.452399    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.456256 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.454801    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.456908 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.456883    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c80431-a853-4012-b239-ea4659c70bf9-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.466857 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.466784    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c80431-a853-4012-b239-ea4659c70bf9-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.467435 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.467409    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfrjx\" (UniqueName: \"kubernetes.io/projected/34c80431-a853-4012-b239-ea4659c70bf9-kube-api-access-hfrjx\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.542405 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.542365    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:58.844836 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.844805    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"]
Nov 03 20:13:58.846653 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:13:58.846618    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34c80431_a853_4012_b239_ea4659c70bf9.slice/crio-86c686a33637de6ca7b42b2f782a7ee085cb8b1c36842cefbe04648d8dbc099e WatchSource:0}: Error finding container 86c686a33637de6ca7b42b2f782a7ee085cb8b1c36842cefbe04648d8dbc099e: Status 404 returned error can't find the container with id 86c686a33637de6ca7b42b2f782a7ee085cb8b1c36842cefbe04648d8dbc099e
Nov 03 20:13:58.849897 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.849850    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:13:58.850000 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.849946    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:13:58.850000 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:58.849987    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:13:59.126674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:59.123447    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z" event={"ID":"34c80431-a853-4012-b239-ea4659c70bf9","Type":"ContainerStarted","Data":"84c1151db91f57fbd6758c31846f3666bc635f87815afe117e56eac0220901db"}
Nov 03 20:13:59.126674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:59.123502    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z" event={"ID":"34c80431-a853-4012-b239-ea4659c70bf9","Type":"ContainerStarted","Data":"86c686a33637de6ca7b42b2f782a7ee085cb8b1c36842cefbe04648d8dbc099e"}
Nov 03 20:13:59.175687 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:59.174610    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z" podStartSLOduration=1.174571262 podStartE2EDuration="1.174571262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:59.160497083 +0000 UTC m=+769.293625546" watchObservedRunningTime="2026-11-03 20:13:59.174571262 +0000 UTC m=+769.307699780"
Nov 03 20:13:59.554621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:59.550962    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:13:59.566733 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:59.565798    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 03 20:13:59.566733 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:13:59.565861    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z" podUID="34c80431-a853-4012-b239-ea4659c70bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 03 20:14:00.551622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:00.544999    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 03 20:14:00.551622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:00.545058    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z" podUID="34c80431-a853-4012-b239-ea4659c70bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 03 20:14:01.560060 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:01.560019    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 03 20:14:01.560705 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:01.560668    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z" podUID="34c80431-a853-4012-b239-ea4659c70bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 03 20:14:02.550041 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:02.549963    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 03 20:14:02.550235 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:02.550037    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z" podUID="34c80431-a853-4012-b239-ea4659c70bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 03 20:14:03.558660 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:03.557998    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 03 20:14:03.558660 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:03.558070    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z" podUID="34c80431-a853-4012-b239-ea4659c70bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 03 20:14:04.545043 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:04.545008    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 03 20:14:04.545337 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:04.545309    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z" podUID="34c80431-a853-4012-b239-ea4659c70bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 03 20:14:06.544373 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:06.544335    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:14:06.544869 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:06.544840    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z" podUID="34c80431-a853-4012-b239-ea4659c70bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:14:07.268061 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:07.268030    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:14:07.268422 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:07.268400    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:14:07.269354 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:07.269331    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:14:17.445448 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:17.445410    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-f97c54-llkmp"]
Nov 03 20:14:17.450930 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:17.450908    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f97c54-llkmp"
Nov 03 20:14:17.459859 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:17.459837    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f97c54-llkmp"]
Nov 03 20:14:17.460120 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:17.460099    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 20:14:17.460510 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:17.460491    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 20:14:17.460682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:17.460525    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-62b0e68a7904\""
Nov 03 20:14:17.460780 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:17.460653    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4a6620c49c4f\""
Nov 03 20:14:17.460780 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:17.460665    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 20:14:17.460780 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:17.460729    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 20:14:17.460919 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:17.460904    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-7ddlc\""
Nov 03 20:14:17.563204 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:17.563174    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-plugins-config-volume\") pod \"payload-processing-external-model-f97c54-llkmp\" (UID: \"e003ddaa-8df6-4700-9ac1-35e3e12e33e7\") " pod="models-as-a-service/payload-processing-external-model-f97c54-llkmp"
Nov 03 20:14:17.563362 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:17.563214    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-provider-credentials\") pod \"payload-processing-external-model-f97c54-llkmp\" (UID: \"e003ddaa-8df6-4700-9ac1-35e3e12e33e7\") " pod="models-as-a-service/payload-processing-external-model-f97c54-llkmp"
Nov 03 20:14:17.563362 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:17.563241    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnpxm\" (UniqueName: \"kubernetes.io/projected/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-kube-api-access-tnpxm\") pod \"payload-processing-external-model-f97c54-llkmp\" (UID: \"e003ddaa-8df6-4700-9ac1-35e3e12e33e7\") " pod="models-as-a-service/payload-processing-external-model-f97c54-llkmp"
Nov 03 20:14:17.563440 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:17.563355    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-routing-overlay-volume\") pod \"payload-processing-external-model-f97c54-llkmp\" (UID: \"e003ddaa-8df6-4700-9ac1-35e3e12e33e7\") " pod="models-as-a-service/payload-processing-external-model-f97c54-llkmp"
Nov 03 20:14:17.664475 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:17.664438    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-plugins-config-volume\") pod \"payload-processing-external-model-f97c54-llkmp\" (UID: \"e003ddaa-8df6-4700-9ac1-35e3e12e33e7\") " pod="models-as-a-service/payload-processing-external-model-f97c54-llkmp"
Nov 03 20:14:17.664798 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:17.664769    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-provider-credentials\") pod \"payload-processing-external-model-f97c54-llkmp\" (UID: \"e003ddaa-8df6-4700-9ac1-35e3e12e33e7\") " pod="models-as-a-service/payload-processing-external-model-f97c54-llkmp"
Nov 03 20:14:17.664992 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:17.664975    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnpxm\" (UniqueName: \"kubernetes.io/projected/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-kube-api-access-tnpxm\") pod \"payload-processing-external-model-f97c54-llkmp\" (UID: \"e003ddaa-8df6-4700-9ac1-35e3e12e33e7\") " pod="models-as-a-service/payload-processing-external-model-f97c54-llkmp"
Nov 03 20:14:17.665129 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:17.665113    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-routing-overlay-volume\") pod \"payload-processing-external-model-f97c54-llkmp\" (UID: \"e003ddaa-8df6-4700-9ac1-35e3e12e33e7\") " pod="models-as-a-service/payload-processing-external-model-f97c54-llkmp"
Nov 03 20:14:17.666906 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:17.666136    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-routing-overlay-volume\") pod \"payload-processing-external-model-f97c54-llkmp\" (UID: \"e003ddaa-8df6-4700-9ac1-35e3e12e33e7\") " pod="models-as-a-service/payload-processing-external-model-f97c54-llkmp"
Nov 03 20:14:17.668420 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:17.668381    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-plugins-config-volume\") pod \"payload-processing-external-model-f97c54-llkmp\" (UID: \"e003ddaa-8df6-4700-9ac1-35e3e12e33e7\") " pod="models-as-a-service/payload-processing-external-model-f97c54-llkmp"
Nov 03 20:14:17.671493 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:17.671452    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-provider-credentials\") pod \"payload-processing-external-model-f97c54-llkmp\" (UID: \"e003ddaa-8df6-4700-9ac1-35e3e12e33e7\") " pod="models-as-a-service/payload-processing-external-model-f97c54-llkmp"
Nov 03 20:14:17.681335 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:17.681302    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnpxm\" (UniqueName: \"kubernetes.io/projected/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-kube-api-access-tnpxm\") pod \"payload-processing-external-model-f97c54-llkmp\" (UID: \"e003ddaa-8df6-4700-9ac1-35e3e12e33e7\") " pod="models-as-a-service/payload-processing-external-model-f97c54-llkmp"
Nov 03 20:14:17.767174 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:17.767073    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f97c54-llkmp"
Nov 03 20:14:18.073276 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:18.073238    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f97c54-llkmp"]
Nov 03 20:14:18.274039 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:18.273997    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f97c54-llkmp" event={"ID":"e003ddaa-8df6-4700-9ac1-35e3e12e33e7","Type":"ContainerStarted","Data":"dafccfb10661df0a825385fd9d4cab70f856841d3d62ebf436cc95ba1a363cbc"}
Nov 03 20:14:18.274267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:18.274071    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f97c54-llkmp" event={"ID":"e003ddaa-8df6-4700-9ac1-35e3e12e33e7","Type":"ContainerStarted","Data":"812c393b204c8571800dd4946660776895d681149d613e9024ad61ceca317689"}
Nov 03 20:14:18.274267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:18.274104    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-f97c54-llkmp"
Nov 03 20:14:18.307583 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:18.307520    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-f97c54-llkmp" podStartSLOduration=1.307498547 podStartE2EDuration="1.307498547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:18.299714054 +0000 UTC m=+788.432842519" watchObservedRunningTime="2026-11-03 20:14:18.307498547 +0000 UTC m=+788.440627010"
Nov 03 20:14:19.639390 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:14:19.639351    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:14:29.322945 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:29.322898    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-f97c54-llkmp"
Nov 03 20:14:30.726282 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:30.725846    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf"]
Nov 03 20:14:30.730397 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:30.730355    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf"]
Nov 03 20:14:30.730550 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:30.730475    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf"
Nov 03 20:14:30.886631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:30.885101    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ea5c483d-5098-4969-8a7c-2d3192e32ea3-provider-credentials\") pod \"payload-processing-external-model-6999b5c985-7x7wf\" (UID: \"ea5c483d-5098-4969-8a7c-2d3192e32ea3\") " pod="models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf"
Nov 03 20:14:30.886631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:30.885178    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swgk6\" (UniqueName: \"kubernetes.io/projected/ea5c483d-5098-4969-8a7c-2d3192e32ea3-kube-api-access-swgk6\") pod \"payload-processing-external-model-6999b5c985-7x7wf\" (UID: \"ea5c483d-5098-4969-8a7c-2d3192e32ea3\") " pod="models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf"
Nov 03 20:14:30.886631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:30.885314    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ea5c483d-5098-4969-8a7c-2d3192e32ea3-routing-overlay-volume\") pod \"payload-processing-external-model-6999b5c985-7x7wf\" (UID: \"ea5c483d-5098-4969-8a7c-2d3192e32ea3\") " pod="models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf"
Nov 03 20:14:30.890643 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:30.888663    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5c483d-5098-4969-8a7c-2d3192e32ea3-plugins-config-volume\") pod \"payload-processing-external-model-6999b5c985-7x7wf\" (UID: \"ea5c483d-5098-4969-8a7c-2d3192e32ea3\") " pod="models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf"
Nov 03 20:14:30.992674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:30.992208    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ea5c483d-5098-4969-8a7c-2d3192e32ea3-provider-credentials\") pod \"payload-processing-external-model-6999b5c985-7x7wf\" (UID: \"ea5c483d-5098-4969-8a7c-2d3192e32ea3\") " pod="models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf"
Nov 03 20:14:30.992674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:30.992270    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swgk6\" (UniqueName: \"kubernetes.io/projected/ea5c483d-5098-4969-8a7c-2d3192e32ea3-kube-api-access-swgk6\") pod \"payload-processing-external-model-6999b5c985-7x7wf\" (UID: \"ea5c483d-5098-4969-8a7c-2d3192e32ea3\") " pod="models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf"
Nov 03 20:14:30.992674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:30.992380    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ea5c483d-5098-4969-8a7c-2d3192e32ea3-routing-overlay-volume\") pod \"payload-processing-external-model-6999b5c985-7x7wf\" (UID: \"ea5c483d-5098-4969-8a7c-2d3192e32ea3\") " pod="models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf"
Nov 03 20:14:30.992674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:30.992498    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5c483d-5098-4969-8a7c-2d3192e32ea3-plugins-config-volume\") pod \"payload-processing-external-model-6999b5c985-7x7wf\" (UID: \"ea5c483d-5098-4969-8a7c-2d3192e32ea3\") " pod="models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf"
Nov 03 20:14:30.993628 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:30.993469    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5c483d-5098-4969-8a7c-2d3192e32ea3-plugins-config-volume\") pod \"payload-processing-external-model-6999b5c985-7x7wf\" (UID: \"ea5c483d-5098-4969-8a7c-2d3192e32ea3\") " pod="models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf"
Nov 03 20:14:31.003818 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:31.003791    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ea5c483d-5098-4969-8a7c-2d3192e32ea3-routing-overlay-volume\") pod \"payload-processing-external-model-6999b5c985-7x7wf\" (UID: \"ea5c483d-5098-4969-8a7c-2d3192e32ea3\") " pod="models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf"
Nov 03 20:14:31.017166 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:31.017112    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ea5c483d-5098-4969-8a7c-2d3192e32ea3-provider-credentials\") pod \"payload-processing-external-model-6999b5c985-7x7wf\" (UID: \"ea5c483d-5098-4969-8a7c-2d3192e32ea3\") " pod="models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf"
Nov 03 20:14:31.022490 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:31.022444    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swgk6\" (UniqueName: \"kubernetes.io/projected/ea5c483d-5098-4969-8a7c-2d3192e32ea3-kube-api-access-swgk6\") pod \"payload-processing-external-model-6999b5c985-7x7wf\" (UID: \"ea5c483d-5098-4969-8a7c-2d3192e32ea3\") " pod="models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf"
Nov 03 20:14:31.130088 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:31.130055    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf"
Nov 03 20:14:31.722329 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:31.719640    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf"]
Nov 03 20:14:32.439108 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:32.439067    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf" event={"ID":"ea5c483d-5098-4969-8a7c-2d3192e32ea3","Type":"ContainerStarted","Data":"789f7d006fc5300f08630cdfc6c49f74fa67762740484948c744eb7091ead1b6"}
Nov 03 20:14:32.439509 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:32.439135    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf" event={"ID":"ea5c483d-5098-4969-8a7c-2d3192e32ea3","Type":"ContainerStarted","Data":"f832a56a604df716b4f1a3b21abcfb91cdc9937c2f44be54240654868a12db2f"}
Nov 03 20:14:32.439509 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:32.439207    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf"
Nov 03 20:14:39.525987 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.525917    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf" podStartSLOduration=9.525892969000001 podStartE2EDuration="9.525892969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:32.47389931 +0000 UTC m=+802.607027772" watchObservedRunningTime="2026-11-03 20:14:39.525892969 +0000 UTC m=+809.659021432"
Nov 03 20:14:39.529098 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.529066    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"]
Nov 03 20:14:39.534183 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.534157    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.556707 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.553027    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-4f93d1-openshift-default-dockercfg-kqdcn\""
Nov 03 20:14:39.566895 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.566869    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"]
Nov 03 20:14:39.671592 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.671553    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.671957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.671931    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.672154 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.672136    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.672287 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.672270    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdhds\" (UniqueName: \"kubernetes.io/projected/9f633998-421c-4d6f-9528-94114a4ca6fe-kube-api-access-mdhds\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.672409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.672394    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.672550 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.672534    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.672709 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.672691    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.672865 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.672848    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f633998-421c-4d6f-9528-94114a4ca6fe-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.673047 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.673028    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.775302 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.775261    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.775571 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.775546    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.775776 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.775755    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdhds\" (UniqueName: \"kubernetes.io/projected/9f633998-421c-4d6f-9528-94114a4ca6fe-kube-api-access-mdhds\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.775914 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.775897    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.776098 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.776048    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.776222 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.776206    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.776379 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.776363    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f633998-421c-4d6f-9528-94114a4ca6fe-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.776552 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.776536    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.776704 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.776685    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.777269 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.777246    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.784625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.778706    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f633998-421c-4d6f-9528-94114a4ca6fe-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.785056 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.785033    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.785737 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.785713    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.786127 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.786106    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.811636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.798067    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.811636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.809148    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.816965 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.816935    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.828273 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.828206    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdhds\" (UniqueName: \"kubernetes.io/projected/9f633998-421c-4d6f-9528-94114a4ca6fe-kube-api-access-mdhds\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:39.880847 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:39.880814    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:40.310115 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:40.310023    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"]
Nov 03 20:14:40.319028 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:14:40.318996    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f633998_421c_4d6f_9528_94114a4ca6fe.slice/crio-4507924bdde3ccb95219bab3364df9c59617aa4d520a08fc71d656b6dfbbd560 WatchSource:0}: Error finding container 4507924bdde3ccb95219bab3364df9c59617aa4d520a08fc71d656b6dfbbd560: Status 404 returned error can't find the container with id 4507924bdde3ccb95219bab3364df9c59617aa4d520a08fc71d656b6dfbbd560
Nov 03 20:14:40.323285 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:40.322417    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:14:40.323285 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:40.322516    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:14:40.323285 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:40.322565    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:14:40.516283 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:40.516251    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng" event={"ID":"9f633998-421c-4d6f-9528-94114a4ca6fe","Type":"ContainerStarted","Data":"199e75df454098a534525ca8c8713ee9e2976670d496c84d58a476a4241fe7b6"}
Nov 03 20:14:40.516482 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:40.516308    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng" event={"ID":"9f633998-421c-4d6f-9528-94114a4ca6fe","Type":"ContainerStarted","Data":"4507924bdde3ccb95219bab3364df9c59617aa4d520a08fc71d656b6dfbbd560"}
Nov 03 20:14:40.540476 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:40.540430    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng" podStartSLOduration=1.5404151750000001 podStartE2EDuration="1.540415175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:40.538103779 +0000 UTC m=+810.671232245" watchObservedRunningTime="2026-11-03 20:14:40.540415175 +0000 UTC m=+810.673543636"
Nov 03 20:14:40.892628 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:40.886284    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:41.883837 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:41.883797    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 20:14:41.884246 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:41.883867    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng" podUID="9f633998-421c-4d6f-9528-94114a4ca6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 20:14:42.881704 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:42.881666    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 20:14:42.881880 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:42.881733    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng" podUID="9f633998-421c-4d6f-9528-94114a4ca6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 20:14:43.295555 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.295523    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"]
Nov 03 20:14:43.300352 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.300325    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.320292 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.320265    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"]
Nov 03 20:14:43.468499 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.468469    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf"
Nov 03 20:14:43.481309 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.481286    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31c0deb-63e5-42a2-a34c-90499e741b2d-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.481423 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.481315    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.481423 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.481337    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.481502 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.481450    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.481545 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.481524    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.481587 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.481572    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.481657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.481619    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpdk8\" (UniqueName: \"kubernetes.io/projected/d31c0deb-63e5-42a2-a34c-90499e741b2d-kube-api-access-tpdk8\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.481737 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.481725    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.481794 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.481779    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.517702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.517674    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f97c54-llkmp"]
Nov 03 20:14:43.583324 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.583240    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpdk8\" (UniqueName: \"kubernetes.io/projected/d31c0deb-63e5-42a2-a34c-90499e741b2d-kube-api-access-tpdk8\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.583473 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.583345    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.583473 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.583409    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.583473 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.583447    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31c0deb-63e5-42a2-a34c-90499e741b2d-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.583643 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.583473    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.583643 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.583534    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.583761 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.583733    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.583839 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.583820    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.583915 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.583900    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.583980 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.583908    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.584293 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.584118    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.584293 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.584187    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.584293 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.584281    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31c0deb-63e5-42a2-a34c-90499e741b2d-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.584634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.584461    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.586639 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.586613    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.586735 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.586716    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.596972 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.596914    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.597571 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.597550    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpdk8\" (UniqueName: \"kubernetes.io/projected/d31c0deb-63e5-42a2-a34c-90499e741b2d-kube-api-access-tpdk8\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.619723 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.619699    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:43.881504 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.881425    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 20:14:43.881504 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.881475    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng" podUID="9f633998-421c-4d6f-9528-94114a4ca6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 20:14:43.936670 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.936640    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"]
Nov 03 20:14:43.937827 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:14:43.937803    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31c0deb_63e5_42a2_a34c_90499e741b2d.slice/crio-c22f644b6da6a0d602d01ff9ca4e9cf3bb4f90d8a3fac6aa21b9d9d0389e9326 WatchSource:0}: Error finding container c22f644b6da6a0d602d01ff9ca4e9cf3bb4f90d8a3fac6aa21b9d9d0389e9326: Status 404 returned error can't find the container with id c22f644b6da6a0d602d01ff9ca4e9cf3bb4f90d8a3fac6aa21b9d9d0389e9326
Nov 03 20:14:43.940293 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.940261    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:14:43.940406 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.940355    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:14:43.940406 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:43.940389    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:14:44.576509 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:44.576468    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn" event={"ID":"d31c0deb-63e5-42a2-a34c-90499e741b2d","Type":"ContainerStarted","Data":"3d7ad5b4d2a608a03587b6ced12f2d778ac20ec3be2a9be4a0f18997bceed0e4"}
Nov 03 20:14:44.577051 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:44.576530    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn" event={"ID":"d31c0deb-63e5-42a2-a34c-90499e741b2d","Type":"ContainerStarted","Data":"c22f644b6da6a0d602d01ff9ca4e9cf3bb4f90d8a3fac6aa21b9d9d0389e9326"}
Nov 03 20:14:44.621690 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:44.619821    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn" podStartSLOduration=1.619802352 podStartE2EDuration="1.619802352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:44.616229461 +0000 UTC m=+814.749357924" watchObservedRunningTime="2026-11-03 20:14:44.619802352 +0000 UTC m=+814.752930814"
Nov 03 20:14:44.621690 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:44.620147    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:44.820618 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:44.820382    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"]
Nov 03 20:14:44.833688 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:44.829930    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"]
Nov 03 20:14:44.833688 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:44.830040    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:14:44.836344 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:44.835722    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f3631f\"/\"distinct-w0-f3631f-kserve-self-signed-certs\""
Nov 03 20:14:44.905825 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:44.905793    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 20:14:44.906091 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:44.906065    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng" podUID="9f633998-421c-4d6f-9528-94114a4ca6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 20:14:44.921381 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:44.921339    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-tmp-dir\") pod \"distinct-w0-f3631f-kserve-7f9ff489c4-67fpl\" (UID: \"5b6f055a-4376-4113-aaaa-335ffbd9091e\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:14:44.921547 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:44.921402    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4csdz\" (UniqueName: \"kubernetes.io/projected/5b6f055a-4376-4113-aaaa-335ffbd9091e-kube-api-access-4csdz\") pod \"distinct-w0-f3631f-kserve-7f9ff489c4-67fpl\" (UID: \"5b6f055a-4376-4113-aaaa-335ffbd9091e\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:14:44.921547 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:44.921458    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-home\") pod \"distinct-w0-f3631f-kserve-7f9ff489c4-67fpl\" (UID: \"5b6f055a-4376-4113-aaaa-335ffbd9091e\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:14:44.921547 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:44.921496    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b6f055a-4376-4113-aaaa-335ffbd9091e-tls-certs\") pod \"distinct-w0-f3631f-kserve-7f9ff489c4-67fpl\" (UID: \"5b6f055a-4376-4113-aaaa-335ffbd9091e\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:14:44.922438 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:44.921666    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-model-cache\") pod \"distinct-w0-f3631f-kserve-7f9ff489c4-67fpl\" (UID: \"5b6f055a-4376-4113-aaaa-335ffbd9091e\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:14:44.923008 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:44.922982    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-dshm\") pod \"distinct-w0-f3631f-kserve-7f9ff489c4-67fpl\" (UID: \"5b6f055a-4376-4113-aaaa-335ffbd9091e\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:14:45.040622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:45.031400    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-dshm\") pod \"distinct-w0-f3631f-kserve-7f9ff489c4-67fpl\" (UID: \"5b6f055a-4376-4113-aaaa-335ffbd9091e\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:14:45.040622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:45.031500    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-tmp-dir\") pod \"distinct-w0-f3631f-kserve-7f9ff489c4-67fpl\" (UID: \"5b6f055a-4376-4113-aaaa-335ffbd9091e\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:14:45.040622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:45.031552    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4csdz\" (UniqueName: \"kubernetes.io/projected/5b6f055a-4376-4113-aaaa-335ffbd9091e-kube-api-access-4csdz\") pod \"distinct-w0-f3631f-kserve-7f9ff489c4-67fpl\" (UID: \"5b6f055a-4376-4113-aaaa-335ffbd9091e\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:14:45.040622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:45.031631    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-home\") pod \"distinct-w0-f3631f-kserve-7f9ff489c4-67fpl\" (UID: \"5b6f055a-4376-4113-aaaa-335ffbd9091e\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:14:45.040622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:45.031671    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b6f055a-4376-4113-aaaa-335ffbd9091e-tls-certs\") pod \"distinct-w0-f3631f-kserve-7f9ff489c4-67fpl\" (UID: \"5b6f055a-4376-4113-aaaa-335ffbd9091e\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:14:45.040622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:45.031715    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-model-cache\") pod \"distinct-w0-f3631f-kserve-7f9ff489c4-67fpl\" (UID: \"5b6f055a-4376-4113-aaaa-335ffbd9091e\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:14:45.040622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:45.032270    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-model-cache\") pod \"distinct-w0-f3631f-kserve-7f9ff489c4-67fpl\" (UID: \"5b6f055a-4376-4113-aaaa-335ffbd9091e\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:14:45.059626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:45.054248    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-dshm\") pod \"distinct-w0-f3631f-kserve-7f9ff489c4-67fpl\" (UID: \"5b6f055a-4376-4113-aaaa-335ffbd9091e\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:14:45.059626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:45.054535    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-tmp-dir\") pod \"distinct-w0-f3631f-kserve-7f9ff489c4-67fpl\" (UID: \"5b6f055a-4376-4113-aaaa-335ffbd9091e\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:14:45.059626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:45.055097    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-home\") pod \"distinct-w0-f3631f-kserve-7f9ff489c4-67fpl\" (UID: \"5b6f055a-4376-4113-aaaa-335ffbd9091e\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:14:45.069273 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:45.069244    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b6f055a-4376-4113-aaaa-335ffbd9091e-tls-certs\") pod \"distinct-w0-f3631f-kserve-7f9ff489c4-67fpl\" (UID: \"5b6f055a-4376-4113-aaaa-335ffbd9091e\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:14:45.096506 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:45.096425    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4csdz\" (UniqueName: \"kubernetes.io/projected/5b6f055a-4376-4113-aaaa-335ffbd9091e-kube-api-access-4csdz\") pod \"distinct-w0-f3631f-kserve-7f9ff489c4-67fpl\" (UID: \"5b6f055a-4376-4113-aaaa-335ffbd9091e\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:14:45.214623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:45.213866    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:14:45.625749 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:45.625651    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:14:45.628023 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:45.627800    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn" podUID="d31c0deb-63e5-42a2-a34c-90499e741b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 20:14:45.803648 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:45.801146    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"]
Nov 03 20:14:45.891888 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:45.881401    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 20:14:45.891888 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:45.881450    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng" podUID="9f633998-421c-4d6f-9528-94114a4ca6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 20:14:46.620162 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:46.620130    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:14:46.620374 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:46.620199    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn" podUID="d31c0deb-63e5-42a2-a34c-90499e741b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 20:14:46.682624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:46.674747    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" event={"ID":"5b6f055a-4376-4113-aaaa-335ffbd9091e","Type":"ContainerStarted","Data":"04ed2b4ac68785febd2e5842a09e13766a49d4f3f6895fcccd82dc98d2bada6d"}
Nov 03 20:14:46.682624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:46.674814    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" event={"ID":"5b6f055a-4376-4113-aaaa-335ffbd9091e","Type":"ContainerStarted","Data":"26815f5311d63a0d2ebb79b44799dab2869ef947939ad9b5af9f3e9366d72d7b"}
Nov 03 20:14:46.888680 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:46.888044    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 20:14:46.888680 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:46.888119    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng" podUID="9f633998-421c-4d6f-9528-94114a4ca6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 20:14:47.214503 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:47.214469    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:14:47.620329 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:47.620241    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:14:47.620329 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:47.620301    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn" podUID="d31c0deb-63e5-42a2-a34c-90499e741b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 20:14:47.886273 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:47.886195    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:47.886644 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:47.886469    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:47.887275 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:47.887257    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:14:47.911663 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:47.911591    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podStartSLOduration=3.911579826 podStartE2EDuration="3.911579826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:46.716540894 +0000 UTC m=+816.849669356" watchObservedRunningTime="2026-11-03 20:14:47.911579826 +0000 UTC m=+818.044708323"
Nov 03 20:14:48.518285 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:48.518244    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-f97c54-llkmp" podUID="e003ddaa-8df6-4700-9ac1-35e3e12e33e7" containerName="payload-processing" containerID="cri-o://dafccfb10661df0a825385fd9d4cab70f856841d3d62ebf436cc95ba1a363cbc" gracePeriod=25
Nov 03 20:14:48.620542 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:48.620513    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:14:48.620707 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:48.620567    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn" podUID="d31c0deb-63e5-42a2-a34c-90499e741b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 20:14:48.691255 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:48.690457    4421 generic.go:356] "Generic (PLEG): container finished" podID="e003ddaa-8df6-4700-9ac1-35e3e12e33e7" containerID="dafccfb10661df0a825385fd9d4cab70f856841d3d62ebf436cc95ba1a363cbc" exitCode=0
Nov 03 20:14:48.691255 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:48.690789    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f97c54-llkmp" event={"ID":"e003ddaa-8df6-4700-9ac1-35e3e12e33e7","Type":"ContainerDied","Data":"dafccfb10661df0a825385fd9d4cab70f856841d3d62ebf436cc95ba1a363cbc"}
Nov 03 20:14:48.839068 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:48.839042    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f97c54-llkmp"
Nov 03 20:14:48.895471 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:48.895437    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-provider-credentials\") pod \"e003ddaa-8df6-4700-9ac1-35e3e12e33e7\" (UID: \"e003ddaa-8df6-4700-9ac1-35e3e12e33e7\") "
Nov 03 20:14:48.895965 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:48.895487    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-routing-overlay-volume\") pod \"e003ddaa-8df6-4700-9ac1-35e3e12e33e7\" (UID: \"e003ddaa-8df6-4700-9ac1-35e3e12e33e7\") "
Nov 03 20:14:48.895965 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:48.895519    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-kube-api-access-tnpxm\" (UniqueName: \"kubernetes.io/projected/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-kube-api-access-tnpxm\") pod \"e003ddaa-8df6-4700-9ac1-35e3e12e33e7\" (UID: \"e003ddaa-8df6-4700-9ac1-35e3e12e33e7\") "
Nov 03 20:14:48.895965 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:48.895921    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-routing-overlay-volume" pod "e003ddaa-8df6-4700-9ac1-35e3e12e33e7" (UID: "e003ddaa-8df6-4700-9ac1-35e3e12e33e7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:14:48.896128 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:48.896097    4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-routing-overlay-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:48.899147 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:48.899116    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-kube-api-access-tnpxm" pod "e003ddaa-8df6-4700-9ac1-35e3e12e33e7" (UID: "e003ddaa-8df6-4700-9ac1-35e3e12e33e7"). InnerVolumeSpecName "kube-api-access-tnpxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:14:48.903865 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:48.903837    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-provider-credentials" pod "e003ddaa-8df6-4700-9ac1-35e3e12e33e7" (UID: "e003ddaa-8df6-4700-9ac1-35e3e12e33e7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:14:48.996518 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:48.996486    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-plugins-config-volume\") pod \"e003ddaa-8df6-4700-9ac1-35e3e12e33e7\" (UID: \"e003ddaa-8df6-4700-9ac1-35e3e12e33e7\") "
Nov 03 20:14:48.996878 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:48.996862    4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-provider-credentials\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:48.996878 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:48.996879    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnpxm\" (UniqueName: \"kubernetes.io/projected/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-kube-api-access-tnpxm\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:48.997018 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:48.996902    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-plugins-config-volume" pod "e003ddaa-8df6-4700-9ac1-35e3e12e33e7" (UID: "e003ddaa-8df6-4700-9ac1-35e3e12e33e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:14:49.098209 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:49.098124    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e003ddaa-8df6-4700-9ac1-35e3e12e33e7-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:49.620916 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:49.620887    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:14:49.621100 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:49.620942    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn" podUID="d31c0deb-63e5-42a2-a34c-90499e741b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 20:14:49.628157 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:14:49.628129    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:14:49.697369 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:49.697336    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f97c54-llkmp" event={"ID":"e003ddaa-8df6-4700-9ac1-35e3e12e33e7","Type":"ContainerDied","Data":"812c393b204c8571800dd4946660776895d681149d613e9024ad61ceca317689"}
Nov 03 20:14:49.697542 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:49.697397    4421 scope.go:122] "RemoveContainer" containerID="dafccfb10661df0a825385fd9d4cab70f856841d3d62ebf436cc95ba1a363cbc"
Nov 03 20:14:49.697542 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:49.697352    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f97c54-llkmp"
Nov 03 20:14:49.720388 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:49.720358    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f97c54-llkmp"]
Nov 03 20:14:49.724221 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:49.724199    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-f97c54-llkmp"]
Nov 03 20:14:50.015497 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:50.015461    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e003ddaa-8df6-4700-9ac1-35e3e12e33e7" path="/var/lib/kubelet/pods/e003ddaa-8df6-4700-9ac1-35e3e12e33e7/volumes"
Nov 03 20:14:50.620279 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:50.620239    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 20:14:50.620577 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:50.620297    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn" podUID="d31c0deb-63e5-42a2-a34c-90499e741b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 20:14:51.853170 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:51.853138    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:51.854089 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:51.854068    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:51.854958 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:51.854934    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:14:55.215369 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:55.215329    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:14:55.228256 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:55.228234    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:14:55.747266 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:55.747235    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:14:59.589762 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:59.581486    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p"]
Nov 03 20:14:59.624290 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:59.624252    4421 cpu_manager.go:414] "containerMap: removing container" podUID="e003ddaa-8df6-4700-9ac1-35e3e12e33e7" containerName="payload-processing"
Nov 03 20:14:59.624545 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:59.624524    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e003ddaa-8df6-4700-9ac1-35e3e12e33e7" containerName="payload-processing"
Nov 03 20:14:59.624927 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:59.624910    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e003ddaa-8df6-4700-9ac1-35e3e12e33e7" containerName="payload-processing"
Nov 03 20:14:59.629657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:59.629634    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p"]
Nov 03 20:14:59.633568 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:59.629915    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p"
Nov 03 20:14:59.643911 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:59.643886    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4a6620c49c4f\""
Nov 03 20:14:59.656783 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:59.656677    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3058d8f4-76e2-4dd7-8b44-04c90b48c722-routing-overlay-volume\") pod \"payload-processing-external-model-9dbdc456b-gcg5p\" (UID: \"3058d8f4-76e2-4dd7-8b44-04c90b48c722\") " pod="models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p"
Nov 03 20:14:59.656783 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:59.656743    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3058d8f4-76e2-4dd7-8b44-04c90b48c722-plugins-config-volume\") pod \"payload-processing-external-model-9dbdc456b-gcg5p\" (UID: \"3058d8f4-76e2-4dd7-8b44-04c90b48c722\") " pod="models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p"
Nov 03 20:14:59.657945 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:59.657919    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr9l9\" (UniqueName: \"kubernetes.io/projected/3058d8f4-76e2-4dd7-8b44-04c90b48c722-kube-api-access-gr9l9\") pod \"payload-processing-external-model-9dbdc456b-gcg5p\" (UID: \"3058d8f4-76e2-4dd7-8b44-04c90b48c722\") " pod="models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p"
Nov 03 20:14:59.658788 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:59.658728    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3058d8f4-76e2-4dd7-8b44-04c90b48c722-provider-credentials\") pod \"payload-processing-external-model-9dbdc456b-gcg5p\" (UID: \"3058d8f4-76e2-4dd7-8b44-04c90b48c722\") " pod="models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p"
Nov 03 20:14:59.759712 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:59.759672    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr9l9\" (UniqueName: \"kubernetes.io/projected/3058d8f4-76e2-4dd7-8b44-04c90b48c722-kube-api-access-gr9l9\") pod \"payload-processing-external-model-9dbdc456b-gcg5p\" (UID: \"3058d8f4-76e2-4dd7-8b44-04c90b48c722\") " pod="models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p"
Nov 03 20:14:59.759913 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:59.759742    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3058d8f4-76e2-4dd7-8b44-04c90b48c722-provider-credentials\") pod \"payload-processing-external-model-9dbdc456b-gcg5p\" (UID: \"3058d8f4-76e2-4dd7-8b44-04c90b48c722\") " pod="models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p"
Nov 03 20:14:59.759913 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:59.759819    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3058d8f4-76e2-4dd7-8b44-04c90b48c722-routing-overlay-volume\") pod \"payload-processing-external-model-9dbdc456b-gcg5p\" (UID: \"3058d8f4-76e2-4dd7-8b44-04c90b48c722\") " pod="models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p"
Nov 03 20:14:59.759913 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:59.759858    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3058d8f4-76e2-4dd7-8b44-04c90b48c722-plugins-config-volume\") pod \"payload-processing-external-model-9dbdc456b-gcg5p\" (UID: \"3058d8f4-76e2-4dd7-8b44-04c90b48c722\") " pod="models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p"
Nov 03 20:14:59.760470 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:59.760449    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3058d8f4-76e2-4dd7-8b44-04c90b48c722-routing-overlay-volume\") pod \"payload-processing-external-model-9dbdc456b-gcg5p\" (UID: \"3058d8f4-76e2-4dd7-8b44-04c90b48c722\") " pod="models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p"
Nov 03 20:14:59.760612 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:59.760579    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3058d8f4-76e2-4dd7-8b44-04c90b48c722-plugins-config-volume\") pod \"payload-processing-external-model-9dbdc456b-gcg5p\" (UID: \"3058d8f4-76e2-4dd7-8b44-04c90b48c722\") " pod="models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p"
Nov 03 20:14:59.762268 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:59.762250    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3058d8f4-76e2-4dd7-8b44-04c90b48c722-provider-credentials\") pod \"payload-processing-external-model-9dbdc456b-gcg5p\" (UID: \"3058d8f4-76e2-4dd7-8b44-04c90b48c722\") " pod="models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p"
Nov 03 20:14:59.772329 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:59.772304    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr9l9\" (UniqueName: \"kubernetes.io/projected/3058d8f4-76e2-4dd7-8b44-04c90b48c722-kube-api-access-gr9l9\") pod \"payload-processing-external-model-9dbdc456b-gcg5p\" (UID: \"3058d8f4-76e2-4dd7-8b44-04c90b48c722\") " pod="models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p"
Nov 03 20:14:59.961056 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:14:59.961029    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p"
Nov 03 20:15:00.152458 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:00.152424    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p"]
Nov 03 20:15:00.157128 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:00.157113    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p"
Nov 03 20:15:00.165677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:00.164819    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a7dfe6-8695-4589-84ad-dc8b844a437f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p\" (UID: \"c3a7dfe6-8695-4589-84ad-dc8b844a437f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p"
Nov 03 20:15:00.165677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:00.164947    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk5mb\" (UniqueName: \"kubernetes.io/projected/c3a7dfe6-8695-4589-84ad-dc8b844a437f-kube-api-access-pk5mb\") pod \"maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p\" (UID: \"c3a7dfe6-8695-4589-84ad-dc8b844a437f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p"
Nov 03 20:15:00.167849 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:00.167821    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\""
Nov 03 20:15:00.171874 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:00.171854    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p"]
Nov 03 20:15:00.240685 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:00.240662    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p"]
Nov 03 20:15:00.242299 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:15:00.242217    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3058d8f4_76e2_4dd7_8b44_04c90b48c722.slice/crio-7c05a388927e7a0ccb716f72bb2655a9ea63bfc5ac94652ea29880a95399710a WatchSource:0}: Error finding container 7c05a388927e7a0ccb716f72bb2655a9ea63bfc5ac94652ea29880a95399710a: Status 404 returned error can't find the container with id 7c05a388927e7a0ccb716f72bb2655a9ea63bfc5ac94652ea29880a95399710a
Nov 03 20:15:00.265876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:00.265846    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk5mb\" (UniqueName: \"kubernetes.io/projected/c3a7dfe6-8695-4589-84ad-dc8b844a437f-kube-api-access-pk5mb\") pod \"maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p\" (UID: \"c3a7dfe6-8695-4589-84ad-dc8b844a437f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p"
Nov 03 20:15:00.265984 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:00.265966    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a7dfe6-8695-4589-84ad-dc8b844a437f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p\" (UID: \"c3a7dfe6-8695-4589-84ad-dc8b844a437f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p"
Nov 03 20:15:00.266615 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:00.266576    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a7dfe6-8695-4589-84ad-dc8b844a437f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p\" (UID: \"c3a7dfe6-8695-4589-84ad-dc8b844a437f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p"
Nov 03 20:15:00.277835 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:00.277812    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk5mb\" (UniqueName: \"kubernetes.io/projected/c3a7dfe6-8695-4589-84ad-dc8b844a437f-kube-api-access-pk5mb\") pod \"maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p\" (UID: \"c3a7dfe6-8695-4589-84ad-dc8b844a437f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p"
Nov 03 20:15:00.477223 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:00.477186    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p"
Nov 03 20:15:00.757138 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:00.757110    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p"]
Nov 03 20:15:00.758410 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:15:00.758387    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a7dfe6_8695_4589_84ad_dc8b844a437f.slice/crio-32b6f6e755a7215a1f2aad842989b8e2843e4a0a90911405f658e2c7f36b0084 WatchSource:0}: Error finding container 32b6f6e755a7215a1f2aad842989b8e2843e4a0a90911405f658e2c7f36b0084: Status 404 returned error can't find the container with id 32b6f6e755a7215a1f2aad842989b8e2843e4a0a90911405f658e2c7f36b0084
Nov 03 20:15:00.788546 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:00.788502    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p" event={"ID":"3058d8f4-76e2-4dd7-8b44-04c90b48c722","Type":"ContainerStarted","Data":"7daf4b23e19f0ef883c3820fdfe80337f4e5f45def67a6b11b9862bc5614c9b6"}
Nov 03 20:15:00.788697 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:00.788580    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p" event={"ID":"3058d8f4-76e2-4dd7-8b44-04c90b48c722","Type":"ContainerStarted","Data":"7c05a388927e7a0ccb716f72bb2655a9ea63bfc5ac94652ea29880a95399710a"}
Nov 03 20:15:00.788769 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:00.788744    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p"
Nov 03 20:15:00.789951 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:00.789925    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p" event={"ID":"c3a7dfe6-8695-4589-84ad-dc8b844a437f","Type":"ContainerStarted","Data":"32b6f6e755a7215a1f2aad842989b8e2843e4a0a90911405f658e2c7f36b0084"}
Nov 03 20:15:00.808730 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:00.808684    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p" podStartSLOduration=1.808673347 podStartE2EDuration="1.808673347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:00.807265795 +0000 UTC m=+830.940394257" watchObservedRunningTime="2026-11-03 20:15:00.808673347 +0000 UTC m=+830.941801808"
Nov 03 20:15:08.971623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:08.968693    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-w9chd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:15:08.971623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:08.968759    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd" podUID="657622a4-6298-4496-b2c3-edce95190d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:10.050389 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.050358    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"]
Nov 03 20:15:10.055035 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.055009    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:15:10.068320 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.068110    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f3631f\"/\"distinct-2-w0-f3631f-kserve-self-signed-certs\""
Nov 03 20:15:10.068774 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.068745    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"]
Nov 03 20:15:10.246185 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.246142    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-model-cache\") pod \"distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg\" (UID: \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:15:10.246359 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.246197    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-home\") pod \"distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg\" (UID: \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:15:10.246359 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.246320    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188223ef-2d2c-4895-9f16-98b1b45aaf6f-tls-certs\") pod \"distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg\" (UID: \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:15:10.246485 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.246367    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-tmp-dir\") pod \"distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg\" (UID: \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:15:10.246547 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.246527    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-dshm\") pod \"distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg\" (UID: \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:15:10.246620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.246580    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjkd9\" (UniqueName: \"kubernetes.io/projected/188223ef-2d2c-4895-9f16-98b1b45aaf6f-kube-api-access-pjkd9\") pod \"distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg\" (UID: \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:15:10.348621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.348335    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-dshm\") pod \"distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg\" (UID: \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:15:10.348621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.348412    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjkd9\" (UniqueName: \"kubernetes.io/projected/188223ef-2d2c-4895-9f16-98b1b45aaf6f-kube-api-access-pjkd9\") pod \"distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg\" (UID: \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:15:10.348621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.348514    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-model-cache\") pod \"distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg\" (UID: \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:15:10.348621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.348555    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-home\") pod \"distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg\" (UID: \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:15:10.349007 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.348632    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188223ef-2d2c-4895-9f16-98b1b45aaf6f-tls-certs\") pod \"distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg\" (UID: \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:15:10.349007 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.348674    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-tmp-dir\") pod \"distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg\" (UID: \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:15:10.352515 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.349555    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-model-cache\") pod \"distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg\" (UID: \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:15:10.352515 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.350196    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-home\") pod \"distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg\" (UID: \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:15:10.352515 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.351565    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-tmp-dir\") pod \"distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg\" (UID: \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:15:10.353976 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.353368    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188223ef-2d2c-4895-9f16-98b1b45aaf6f-tls-certs\") pod \"distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg\" (UID: \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:15:10.356040 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.356013    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-dshm\") pod \"distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg\" (UID: \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:15:10.363900 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.363875    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjkd9\" (UniqueName: \"kubernetes.io/projected/188223ef-2d2c-4895-9f16-98b1b45aaf6f-kube-api-access-pjkd9\") pod \"distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg\" (UID: \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\") " pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:15:10.371768 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.371737    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:15:10.669700 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.669676    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"]
Nov 03 20:15:10.693496 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:15:10.693464    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod188223ef_2d2c_4895_9f16_98b1b45aaf6f.slice/crio-0adddaf540953be7ec4a143b6a405bea39fd0da48cb6742a7aaefb0eda938f83 WatchSource:0}: Error finding container 0adddaf540953be7ec4a143b6a405bea39fd0da48cb6742a7aaefb0eda938f83: Status 404 returned error can't find the container with id 0adddaf540953be7ec4a143b6a405bea39fd0da48cb6742a7aaefb0eda938f83
Nov 03 20:15:10.945066 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.944979    4421 generic.go:356] "Generic (PLEG): container finished" podID="c3a7dfe6-8695-4589-84ad-dc8b844a437f" containerID="2366bb1ef1766bb39a393aa33e829fdf8edb07e06cfcecf418d21da071dc1409" exitCode=60
Nov 03 20:15:10.945234 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.945057    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p" event={"ID":"c3a7dfe6-8695-4589-84ad-dc8b844a437f","Type":"ContainerDied","Data":"2366bb1ef1766bb39a393aa33e829fdf8edb07e06cfcecf418d21da071dc1409"}
Nov 03 20:15:10.945334 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.945318    4421 scope.go:122] "RemoveContainer" containerID="2366bb1ef1766bb39a393aa33e829fdf8edb07e06cfcecf418d21da071dc1409"
Nov 03 20:15:10.947004 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.946877    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" event={"ID":"188223ef-2d2c-4895-9f16-98b1b45aaf6f","Type":"ContainerStarted","Data":"91504c45ebd819efa3f91cfc8a06dddec47324d63a55f46a862bdcb126d74f0c"}
Nov 03 20:15:10.947004 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.946909    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" event={"ID":"188223ef-2d2c-4895-9f16-98b1b45aaf6f","Type":"ContainerStarted","Data":"0adddaf540953be7ec4a143b6a405bea39fd0da48cb6742a7aaefb0eda938f83"}
Nov 03 20:15:10.984854 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:10.984811    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podStartSLOduration=0.984796527 podStartE2EDuration="984.796527ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:10.984319661 +0000 UTC m=+841.117448124" watchObservedRunningTime="2026-11-03 20:15:10.984796527 +0000 UTC m=+841.117924988"
Nov 03 20:15:11.372851 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:11.372820    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:15:11.798522 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:11.798496    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p"
Nov 03 20:15:11.866636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:11.857085    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf"]
Nov 03 20:15:11.954965 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:11.954931    4421 generic.go:356] "Generic (PLEG): container finished" podID="c3a7dfe6-8695-4589-84ad-dc8b844a437f" containerID="27f2a133d5b7223e817a1d0b22a36edce8684769db83ee76dca93269b70de626" exitCode=60
Nov 03 20:15:11.955150 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:11.955009    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p" event={"ID":"c3a7dfe6-8695-4589-84ad-dc8b844a437f","Type":"ContainerDied","Data":"27f2a133d5b7223e817a1d0b22a36edce8684769db83ee76dca93269b70de626"}
Nov 03 20:15:11.955150 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:11.955071    4421 scope.go:122] "RemoveContainer" containerID="2366bb1ef1766bb39a393aa33e829fdf8edb07e06cfcecf418d21da071dc1409"
Nov 03 20:15:11.955538 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:11.955518    4421 scope.go:122] "RemoveContainer" containerID="27f2a133d5b7223e817a1d0b22a36edce8684769db83ee76dca93269b70de626"
Nov 03 20:15:11.955825 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:15:11.955799    4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p_odh-ai-gateway-infra(c3a7dfe6-8695-4589-84ad-dc8b844a437f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p" podUID="c3a7dfe6-8695-4589-84ad-dc8b844a437f"
Nov 03 20:15:12.962569 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:12.962539    4421 scope.go:122] "RemoveContainer" containerID="27f2a133d5b7223e817a1d0b22a36edce8684769db83ee76dca93269b70de626"
Nov 03 20:15:12.963092 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:15:12.962810    4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p_odh-ai-gateway-infra(c3a7dfe6-8695-4589-84ad-dc8b844a437f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p" podUID="c3a7dfe6-8695-4589-84ad-dc8b844a437f"
Nov 03 20:15:16.858011 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:16.857965    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf" podUID="ea5c483d-5098-4969-8a7c-2d3192e32ea3" containerName="payload-processing" containerID="cri-o://789f7d006fc5300f08630cdfc6c49f74fa67762740484948c744eb7091ead1b6" gracePeriod=25
Nov 03 20:15:16.992852 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:16.992817    4421 generic.go:356] "Generic (PLEG): container finished" podID="ea5c483d-5098-4969-8a7c-2d3192e32ea3" containerID="789f7d006fc5300f08630cdfc6c49f74fa67762740484948c744eb7091ead1b6" exitCode=0
Nov 03 20:15:16.993029 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:16.992905    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf" event={"ID":"ea5c483d-5098-4969-8a7c-2d3192e32ea3","Type":"ContainerDied","Data":"789f7d006fc5300f08630cdfc6c49f74fa67762740484948c744eb7091ead1b6"}
Nov 03 20:15:17.149340 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:17.149315    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf"
Nov 03 20:15:17.251556 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:17.251526    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea5c483d-5098-4969-8a7c-2d3192e32ea3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5c483d-5098-4969-8a7c-2d3192e32ea3-plugins-config-volume\") pod \"ea5c483d-5098-4969-8a7c-2d3192e32ea3\" (UID: \"ea5c483d-5098-4969-8a7c-2d3192e32ea3\") "
Nov 03 20:15:17.251780 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:17.251590    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea5c483d-5098-4969-8a7c-2d3192e32ea3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ea5c483d-5098-4969-8a7c-2d3192e32ea3-routing-overlay-volume\") pod \"ea5c483d-5098-4969-8a7c-2d3192e32ea3\" (UID: \"ea5c483d-5098-4969-8a7c-2d3192e32ea3\") "
Nov 03 20:15:17.251780 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:17.251729    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea5c483d-5098-4969-8a7c-2d3192e32ea3-kube-api-access-swgk6\" (UniqueName: \"kubernetes.io/projected/ea5c483d-5098-4969-8a7c-2d3192e32ea3-kube-api-access-swgk6\") pod \"ea5c483d-5098-4969-8a7c-2d3192e32ea3\" (UID: \"ea5c483d-5098-4969-8a7c-2d3192e32ea3\") "
Nov 03 20:15:17.251918 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:17.251903    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea5c483d-5098-4969-8a7c-2d3192e32ea3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ea5c483d-5098-4969-8a7c-2d3192e32ea3-provider-credentials\") pod \"ea5c483d-5098-4969-8a7c-2d3192e32ea3\" (UID: \"ea5c483d-5098-4969-8a7c-2d3192e32ea3\") "
Nov 03 20:15:17.251973 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:17.251922    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea5c483d-5098-4969-8a7c-2d3192e32ea3-plugins-config-volume" pod "ea5c483d-5098-4969-8a7c-2d3192e32ea3" (UID: "ea5c483d-5098-4969-8a7c-2d3192e32ea3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:17.251973 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:17.251944    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea5c483d-5098-4969-8a7c-2d3192e32ea3-routing-overlay-volume" pod "ea5c483d-5098-4969-8a7c-2d3192e32ea3" (UID: "ea5c483d-5098-4969-8a7c-2d3192e32ea3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:17.252281 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:17.252265    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5c483d-5098-4969-8a7c-2d3192e32ea3-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:17.252348 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:17.252286    4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ea5c483d-5098-4969-8a7c-2d3192e32ea3-routing-overlay-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:17.253970 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:17.253946    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5c483d-5098-4969-8a7c-2d3192e32ea3-kube-api-access-swgk6" pod "ea5c483d-5098-4969-8a7c-2d3192e32ea3" (UID: "ea5c483d-5098-4969-8a7c-2d3192e32ea3"). InnerVolumeSpecName "kube-api-access-swgk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:17.254022 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:17.254000    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5c483d-5098-4969-8a7c-2d3192e32ea3-provider-credentials" pod "ea5c483d-5098-4969-8a7c-2d3192e32ea3" (UID: "ea5c483d-5098-4969-8a7c-2d3192e32ea3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:17.362093 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:17.362018    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swgk6\" (UniqueName: \"kubernetes.io/projected/ea5c483d-5098-4969-8a7c-2d3192e32ea3-kube-api-access-swgk6\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:17.362093 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:17.362060    4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ea5c483d-5098-4969-8a7c-2d3192e32ea3-provider-credentials\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:18.000203 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:18.000173    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf"
Nov 03 20:15:18.000650 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:18.000173    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf" event={"ID":"ea5c483d-5098-4969-8a7c-2d3192e32ea3","Type":"ContainerDied","Data":"f832a56a604df716b4f1a3b21abcfb91cdc9937c2f44be54240654868a12db2f"}
Nov 03 20:15:18.000650 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:18.000304    4421 scope.go:122] "RemoveContainer" containerID="789f7d006fc5300f08630cdfc6c49f74fa67762740484948c744eb7091ead1b6"
Nov 03 20:15:18.025139 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:18.025114    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf"]
Nov 03 20:15:18.029682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:18.029658    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6999b5c985-7x7wf"]
Nov 03 20:15:19.628766 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:15:19.628690    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:15:20.015260 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:20.015228    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea5c483d-5098-4969-8a7c-2d3192e32ea3" path="/var/lib/kubelet/pods/ea5c483d-5098-4969-8a7c-2d3192e32ea3/volumes"
Nov 03 20:15:20.372617 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:20.372515    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:15:20.385700 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:20.385677    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:15:21.034142 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:21.034107    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:15:24.603651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.603617    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"]
Nov 03 20:15:24.604149 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.604136    4421 cpu_manager.go:414] "containerMap: removing container" podUID="ea5c483d-5098-4969-8a7c-2d3192e32ea3" containerName="payload-processing"
Nov 03 20:15:24.604192 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.604150    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea5c483d-5098-4969-8a7c-2d3192e32ea3" containerName="payload-processing"
Nov 03 20:15:24.604226 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.604219    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea5c483d-5098-4969-8a7c-2d3192e32ea3" containerName="payload-processing"
Nov 03 20:15:24.607467 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.607449    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.621695 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.621671    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"]
Nov 03 20:15:24.787084 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.787041    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsdrw\" (UniqueName: \"kubernetes.io/projected/7eefc668-8d72-4ee2-b047-91d13318ec49-kube-api-access-tsdrw\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.799170 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.799136    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.799440 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.799416    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.799786 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.799757    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.799892 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.799838    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.799892 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.799866    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.800019 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.799909    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.800019 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.799957    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eefc668-8d72-4ee2-b047-91d13318ec49-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.800190 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.800055    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.901341 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.901260    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.901341 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.901294    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eefc668-8d72-4ee2-b047-91d13318ec49-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.901341 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.901315    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.901656 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.901379    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsdrw\" (UniqueName: \"kubernetes.io/projected/7eefc668-8d72-4ee2-b047-91d13318ec49-kube-api-access-tsdrw\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.901656 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.901419    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.901656 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.901547    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.901907 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.901659    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.901907 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.901719    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.901907 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.901746    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.901907 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.901837    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.902067 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.902042    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.902130 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.902093    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.902130 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.902108    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.902244 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.902128    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eefc668-8d72-4ee2-b047-91d13318ec49-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.904149 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.904128    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.904269 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.904201    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.913551 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.913531    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.913690 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.913673    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsdrw\" (UniqueName: \"kubernetes.io/projected/7eefc668-8d72-4ee2-b047-91d13318ec49-kube-api-access-tsdrw\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:24.922731 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:24.922615    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:25.206834 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:25.206807    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"]
Nov 03 20:15:25.208152 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:15:25.208123    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eefc668_8d72_4ee2_b047_91d13318ec49.slice/crio-673af302896cae623bf6a3dbac75768776ba55d92075ca9f7eaa336e77bfec86 WatchSource:0}: Error finding container 673af302896cae623bf6a3dbac75768776ba55d92075ca9f7eaa336e77bfec86: Status 404 returned error can't find the container with id 673af302896cae623bf6a3dbac75768776ba55d92075ca9f7eaa336e77bfec86
Nov 03 20:15:25.210668 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:25.210635    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:15:25.210751 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:25.210699    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:15:25.210751 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:25.210728    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:15:26.010224 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:26.010184    4421 scope.go:122] "RemoveContainer" containerID="27f2a133d5b7223e817a1d0b22a36edce8684769db83ee76dca93269b70de626"
Nov 03 20:15:26.102317 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:26.102289    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9" event={"ID":"7eefc668-8d72-4ee2-b047-91d13318ec49","Type":"ContainerStarted","Data":"b01881ec6dfe12f1dbc37fff50f48ad859e9531b5e004f87d0c831acae2a64dd"}
Nov 03 20:15:26.102444 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:26.102338    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9" event={"ID":"7eefc668-8d72-4ee2-b047-91d13318ec49","Type":"ContainerStarted","Data":"673af302896cae623bf6a3dbac75768776ba55d92075ca9f7eaa336e77bfec86"}
Nov 03 20:15:26.131906 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:26.131863    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9" podStartSLOduration=2.131849517 podStartE2EDuration="2.131849517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:26.128642633 +0000 UTC m=+856.261771095" watchObservedRunningTime="2026-11-03 20:15:26.131849517 +0000 UTC m=+856.264978051"
Nov 03 20:15:26.922919 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:26.922882    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:26.924528 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:26.924503    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:15:26.924681 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:26.924550    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9" podUID="7eefc668-8d72-4ee2-b047-91d13318ec49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 20:15:27.109906 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:27.109871    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p" event={"ID":"c3a7dfe6-8695-4589-84ad-dc8b844a437f","Type":"ContainerStarted","Data":"722e4d813b592c5fb1b48ab8a237f546f96c60a26bc23074a2996ef50472cb65"}
Nov 03 20:15:27.129304 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:27.129260    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p" podStartSLOduration=18.290115168 podStartE2EDuration="27.129247436s" totalImagesPullingTime="8.839132268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:27.127463497 +0000 UTC m=+857.260591960" watchObservedRunningTime="2026-11-03 20:15:27.129247436 +0000 UTC m=+857.262375898"
Nov 03 20:15:27.923502 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:27.923467    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:15:27.923706 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:27.923532    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9" podUID="7eefc668-8d72-4ee2-b047-91d13318ec49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 20:15:28.117523 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.117488    4421 generic.go:356] "Generic (PLEG): container finished" podID="c3a7dfe6-8695-4589-84ad-dc8b844a437f" containerID="722e4d813b592c5fb1b48ab8a237f546f96c60a26bc23074a2996ef50472cb65" exitCode=60
Nov 03 20:15:28.118243 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.117571    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p" event={"ID":"c3a7dfe6-8695-4589-84ad-dc8b844a437f","Type":"ContainerDied","Data":"722e4d813b592c5fb1b48ab8a237f546f96c60a26bc23074a2996ef50472cb65"}
Nov 03 20:15:28.118243 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.117658    4421 scope.go:122] "RemoveContainer" containerID="27f2a133d5b7223e817a1d0b22a36edce8684769db83ee76dca93269b70de626"
Nov 03 20:15:28.118243 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.118184    4421 scope.go:122] "RemoveContainer" containerID="722e4d813b592c5fb1b48ab8a237f546f96c60a26bc23074a2996ef50472cb65"
Nov 03 20:15:28.118543 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:15:28.118458    4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p_odh-ai-gateway-infra(c3a7dfe6-8695-4589-84ad-dc8b844a437f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p" podUID="c3a7dfe6-8695-4589-84ad-dc8b844a437f"
Nov 03 20:15:28.185787 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.185670    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p"]
Nov 03 20:15:28.418582 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.418553    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"]
Nov 03 20:15:28.424303 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.424272    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.438088 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.437691    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"]
Nov 03 20:15:28.480953 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.480899    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.481156 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.480984    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf17e0a-2b24-4633-afb7-44eebab411d3-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.481156 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.481094    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.481156 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.481123    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmqx5\" (UniqueName: \"kubernetes.io/projected/3cf17e0a-2b24-4633-afb7-44eebab411d3-kube-api-access-vmqx5\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.481319 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.481261    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.481377 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.481347    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.481473 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.481433    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.481571 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.481554    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.481643 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.481586    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.593624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.585521    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.593624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.585613    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf17e0a-2b24-4633-afb7-44eebab411d3-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.593624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.585659    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.593624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.585702    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmqx5\" (UniqueName: \"kubernetes.io/projected/3cf17e0a-2b24-4633-afb7-44eebab411d3-kube-api-access-vmqx5\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.593624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.585764    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.593624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.585841    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.593624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.585880    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.593624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.585940    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.593624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.585965    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.593624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.588562    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.593624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.589481    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.593624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.590749    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf17e0a-2b24-4633-afb7-44eebab411d3-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.593624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.590923    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.593624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.591250    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.624622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.618703    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.635630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.625283    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.635630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.625629    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.635630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.626220    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmqx5\" (UniqueName: \"kubernetes.io/projected/3cf17e0a-2b24-4633-afb7-44eebab411d3-kube-api-access-vmqx5\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.785674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.779734    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:28.923526 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.923470    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:15:28.923709 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:28.923584    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9" podUID="7eefc668-8d72-4ee2-b047-91d13318ec49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 20:15:29.276221 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:29.275981    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"]
Nov 03 20:15:29.276896 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:15:29.276867    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf17e0a_2b24_4633_afb7_44eebab411d3.slice/crio-85a14b9558e5084c6c14441456302ebf7a3e4f8bda7bc59b30080ec274ba8e8c WatchSource:0}: Error finding container 85a14b9558e5084c6c14441456302ebf7a3e4f8bda7bc59b30080ec274ba8e8c: Status 404 returned error can't find the container with id 85a14b9558e5084c6c14441456302ebf7a3e4f8bda7bc59b30080ec274ba8e8c
Nov 03 20:15:29.279889 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:29.279853    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:15:29.279987 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:29.279935    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:15:29.279987 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:29.279975    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:15:29.300187 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:29.300168    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p"
Nov 03 20:15:29.316644 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:29.316592    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3a7dfe6-8695-4589-84ad-dc8b844a437f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a7dfe6-8695-4589-84ad-dc8b844a437f-maas-api-service-ca\") pod \"c3a7dfe6-8695-4589-84ad-dc8b844a437f\" (UID: \"c3a7dfe6-8695-4589-84ad-dc8b844a437f\") "
Nov 03 20:15:29.316753 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:29.316715    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3a7dfe6-8695-4589-84ad-dc8b844a437f-kube-api-access-pk5mb\" (UniqueName: \"kubernetes.io/projected/c3a7dfe6-8695-4589-84ad-dc8b844a437f-kube-api-access-pk5mb\") pod \"c3a7dfe6-8695-4589-84ad-dc8b844a437f\" (UID: \"c3a7dfe6-8695-4589-84ad-dc8b844a437f\") "
Nov 03 20:15:29.316923 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:29.316897    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a7dfe6-8695-4589-84ad-dc8b844a437f-maas-api-service-ca" pod "c3a7dfe6-8695-4589-84ad-dc8b844a437f" (UID: "c3a7dfe6-8695-4589-84ad-dc8b844a437f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:29.317170 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:29.317146    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3a7dfe6-8695-4589-84ad-dc8b844a437f-maas-api-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:29.320977 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:29.320938    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a7dfe6-8695-4589-84ad-dc8b844a437f-kube-api-access-pk5mb" pod "c3a7dfe6-8695-4589-84ad-dc8b844a437f" (UID: "c3a7dfe6-8695-4589-84ad-dc8b844a437f"). InnerVolumeSpecName "kube-api-access-pk5mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:29.418086 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:29.418044    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk5mb\" (UniqueName: \"kubernetes.io/projected/c3a7dfe6-8695-4589-84ad-dc8b844a437f-kube-api-access-pk5mb\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:29.923945 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:29.923905    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:15:29.924129 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:29.923964    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9" podUID="7eefc668-8d72-4ee2-b047-91d13318ec49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 20:15:30.161775 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:30.161722    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7" event={"ID":"3cf17e0a-2b24-4633-afb7-44eebab411d3","Type":"ContainerStarted","Data":"60856f4ccb96c1b6dfbc9a8dea5e05dff094720bc3b796e039dbc3a8e57c13cf"}
Nov 03 20:15:30.162058 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:30.162034    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7" event={"ID":"3cf17e0a-2b24-4633-afb7-44eebab411d3","Type":"ContainerStarted","Data":"85a14b9558e5084c6c14441456302ebf7a3e4f8bda7bc59b30080ec274ba8e8c"}
Nov 03 20:15:30.165695 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:30.163708    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p" event={"ID":"c3a7dfe6-8695-4589-84ad-dc8b844a437f","Type":"ContainerDied","Data":"32b6f6e755a7215a1f2aad842989b8e2843e4a0a90911405f658e2c7f36b0084"}
Nov 03 20:15:30.165695 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:30.163761    4421 scope.go:122] "RemoveContainer" containerID="722e4d813b592c5fb1b48ab8a237f546f96c60a26bc23074a2996ef50472cb65"
Nov 03 20:15:30.165695 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:30.163878    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p"
Nov 03 20:15:30.201289 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:30.200442    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7" podStartSLOduration=2.200420449 podStartE2EDuration="2.200420449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:30.192139946 +0000 UTC m=+860.325268408" watchObservedRunningTime="2026-11-03 20:15:30.200420449 +0000 UTC m=+860.333548914"
Nov 03 20:15:30.209416 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:30.209385    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p"]
Nov 03 20:15:30.213885 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:30.213860    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895615-6lm4p"]
Nov 03 20:15:30.800682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:30.800426    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:30.800682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:30.800566    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:15:30.800682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:30.800647    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7" podUID="3cf17e0a-2b24-4633-afb7-44eebab411d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 20:15:30.923562 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:30.923531    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 20:15:30.923741 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:30.923613    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9" podUID="7eefc668-8d72-4ee2-b047-91d13318ec49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 20:15:31.794825 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:31.794787    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:15:31.795007 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:31.794852    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7" podUID="3cf17e0a-2b24-4633-afb7-44eebab411d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 20:15:32.036542 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:32.036157    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3a7dfe6-8695-4589-84ad-dc8b844a437f" path="/var/lib/kubelet/pods/c3a7dfe6-8695-4589-84ad-dc8b844a437f/volumes"
Nov 03 20:15:32.173258 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:32.172131    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:32.173258 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:32.172639    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:32.173888 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:32.173851    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:15:32.808007 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:32.807965    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:15:32.808195 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:32.808030    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7" podUID="3cf17e0a-2b24-4633-afb7-44eebab411d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 20:15:33.794291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:33.794252    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:15:33.794799 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:33.794317    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7" podUID="3cf17e0a-2b24-4633-afb7-44eebab411d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 20:15:34.793959 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:34.793928    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:15:34.794155 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:34.793980    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7" podUID="3cf17e0a-2b24-4633-afb7-44eebab411d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 20:15:35.794139 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:35.794101    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 20:15:35.794652 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:35.794179    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7" podUID="3cf17e0a-2b24-4633-afb7-44eebab411d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 20:15:37.790625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:37.789865    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:37.814626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:37.808691    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:15:37.814626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:37.808748    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7" podUID="3cf17e0a-2b24-4633-afb7-44eebab411d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:37.947625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:37.943928    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:38.323624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:38.322688    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:15:38.323624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:38.322747    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z" podUID="34c80431-a853-4012-b239-ea4659c70bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:38.737741 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:38.737694    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:38.789724 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:38.789679    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:38.810721 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:38.810683    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:15:38.811307 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:38.811275    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7" podUID="3cf17e0a-2b24-4633-afb7-44eebab411d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:38.824914 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:38.824888    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:38.825738 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:38.825718    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:38.835838 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:38.835816    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:15:38.946135 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:38.946093    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:38.947373 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:38.947350    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:15:39.069681 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:39.069340    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:39.746071 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:39.746028    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:15:39.746991 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:39.746966    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:15:39.816651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:39.816590    4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:39.817307 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:39.817279    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 03 20:15:39.818227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:39.818210    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:15:39.846541 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:39.816520    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:15:39.979895 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:39.979858    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 20:15:40.803622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:40.738985    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:15:40.831617 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:40.831559    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:40.915646 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:40.915581    4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:41.199049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:41.014323    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:41.199327 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:41.051858    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:15:41.199787 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:41.139810    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:15:41.380886 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:41.380860    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:15:41.459273 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:41.459183    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p"]
Nov 03 20:15:46.467484 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:46.467441    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p" podUID="3058d8f4-76e2-4dd7-8b44-04c90b48c722" containerName="payload-processing" containerID="cri-o://7daf4b23e19f0ef883c3820fdfe80337f4e5f45def67a6b11b9862bc5614c9b6" gracePeriod=25
Nov 03 20:15:46.792442 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:46.792414    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p"
Nov 03 20:15:46.943670 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:46.943637    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3058d8f4-76e2-4dd7-8b44-04c90b48c722-kube-api-access-gr9l9\" (UniqueName: \"kubernetes.io/projected/3058d8f4-76e2-4dd7-8b44-04c90b48c722-kube-api-access-gr9l9\") pod \"3058d8f4-76e2-4dd7-8b44-04c90b48c722\" (UID: \"3058d8f4-76e2-4dd7-8b44-04c90b48c722\") "
Nov 03 20:15:46.943862 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:46.943710    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3058d8f4-76e2-4dd7-8b44-04c90b48c722-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3058d8f4-76e2-4dd7-8b44-04c90b48c722-provider-credentials\") pod \"3058d8f4-76e2-4dd7-8b44-04c90b48c722\" (UID: \"3058d8f4-76e2-4dd7-8b44-04c90b48c722\") "
Nov 03 20:15:46.943862 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:46.943758    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3058d8f4-76e2-4dd7-8b44-04c90b48c722-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3058d8f4-76e2-4dd7-8b44-04c90b48c722-routing-overlay-volume\") pod \"3058d8f4-76e2-4dd7-8b44-04c90b48c722\" (UID: \"3058d8f4-76e2-4dd7-8b44-04c90b48c722\") "
Nov 03 20:15:46.943862 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:46.943828    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3058d8f4-76e2-4dd7-8b44-04c90b48c722-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3058d8f4-76e2-4dd7-8b44-04c90b48c722-plugins-config-volume\") pod \"3058d8f4-76e2-4dd7-8b44-04c90b48c722\" (UID: \"3058d8f4-76e2-4dd7-8b44-04c90b48c722\") "
Nov 03 20:15:46.944160 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:46.944124    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3058d8f4-76e2-4dd7-8b44-04c90b48c722-routing-overlay-volume" pod "3058d8f4-76e2-4dd7-8b44-04c90b48c722" (UID: "3058d8f4-76e2-4dd7-8b44-04c90b48c722"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:46.944301 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:46.944243    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3058d8f4-76e2-4dd7-8b44-04c90b48c722-plugins-config-volume" pod "3058d8f4-76e2-4dd7-8b44-04c90b48c722" (UID: "3058d8f4-76e2-4dd7-8b44-04c90b48c722"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:46.946085 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:46.946062    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3058d8f4-76e2-4dd7-8b44-04c90b48c722-kube-api-access-gr9l9" pod "3058d8f4-76e2-4dd7-8b44-04c90b48c722" (UID: "3058d8f4-76e2-4dd7-8b44-04c90b48c722"). InnerVolumeSpecName "kube-api-access-gr9l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:46.946181 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:46.946094    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3058d8f4-76e2-4dd7-8b44-04c90b48c722-provider-credentials" pod "3058d8f4-76e2-4dd7-8b44-04c90b48c722" (UID: "3058d8f4-76e2-4dd7-8b44-04c90b48c722"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:47.045219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:47.045138    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr9l9\" (UniqueName: \"kubernetes.io/projected/3058d8f4-76e2-4dd7-8b44-04c90b48c722-kube-api-access-gr9l9\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:47.045219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:47.045166    4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3058d8f4-76e2-4dd7-8b44-04c90b48c722-provider-credentials\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:47.045219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:47.045177    4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3058d8f4-76e2-4dd7-8b44-04c90b48c722-routing-overlay-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:47.045219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:47.045191    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3058d8f4-76e2-4dd7-8b44-04c90b48c722-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:47.147669 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:47.147630    4421 generic.go:356] "Generic (PLEG): container finished" podID="3058d8f4-76e2-4dd7-8b44-04c90b48c722" containerID="7daf4b23e19f0ef883c3820fdfe80337f4e5f45def67a6b11b9862bc5614c9b6" exitCode=0
Nov 03 20:15:47.147864 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:47.147751    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p"
Nov 03 20:15:47.147864 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:47.147775    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p" event={"ID":"3058d8f4-76e2-4dd7-8b44-04c90b48c722","Type":"ContainerDied","Data":"7daf4b23e19f0ef883c3820fdfe80337f4e5f45def67a6b11b9862bc5614c9b6"}
Nov 03 20:15:47.147864 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:47.147838    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p" event={"ID":"3058d8f4-76e2-4dd7-8b44-04c90b48c722","Type":"ContainerDied","Data":"7c05a388927e7a0ccb716f72bb2655a9ea63bfc5ac94652ea29880a95399710a"}
Nov 03 20:15:47.148033 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:47.147867    4421 scope.go:122] "RemoveContainer" containerID="7daf4b23e19f0ef883c3820fdfe80337f4e5f45def67a6b11b9862bc5614c9b6"
Nov 03 20:15:47.157943 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:47.157921    4421 scope.go:122] "RemoveContainer" containerID="7daf4b23e19f0ef883c3820fdfe80337f4e5f45def67a6b11b9862bc5614c9b6"
Nov 03 20:15:47.158205 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:15:47.158185    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7daf4b23e19f0ef883c3820fdfe80337f4e5f45def67a6b11b9862bc5614c9b6\": container with ID starting with 7daf4b23e19f0ef883c3820fdfe80337f4e5f45def67a6b11b9862bc5614c9b6 not found: ID does not exist" containerID="7daf4b23e19f0ef883c3820fdfe80337f4e5f45def67a6b11b9862bc5614c9b6"
Nov 03 20:15:47.158251 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:47.158214    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7daf4b23e19f0ef883c3820fdfe80337f4e5f45def67a6b11b9862bc5614c9b6"} err="failed to get container status \"7daf4b23e19f0ef883c3820fdfe80337f4e5f45def67a6b11b9862bc5614c9b6\": rpc error: code = NotFound desc = could not find container \"7daf4b23e19f0ef883c3820fdfe80337f4e5f45def67a6b11b9862bc5614c9b6\": container with ID starting with 7daf4b23e19f0ef883c3820fdfe80337f4e5f45def67a6b11b9862bc5614c9b6 not found: ID does not exist"
Nov 03 20:15:47.173670 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:47.173647    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p"]
Nov 03 20:15:47.177587 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:47.177568    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-9dbdc456b-gcg5p"]
Nov 03 20:15:48.018501 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:48.018471    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3058d8f4-76e2-4dd7-8b44-04c90b48c722" path="/var/lib/kubelet/pods/3058d8f4-76e2-4dd7-8b44-04c90b48c722/volumes"
Nov 03 20:15:49.659050 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:15:49.659008    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:15:55.088729 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:55.088678    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:15:56.031547 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:56.030543    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:56.089811 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:56.089771    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:56.091072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:56.091047    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:15:56.328198 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:56.327994    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:15:57.523623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:57.521690    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-6bhmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:15:57.523623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:57.521798    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr" podUID="9d6e2ee7-7d48-4879-b6fa-f303b5d24697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:15:59.525289 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.525257    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"]
Nov 03 20:15:59.525918 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.525901    4421 cpu_manager.go:414] "containerMap: removing container" podUID="c3a7dfe6-8695-4589-84ad-dc8b844a437f" containerName="cleanup"
Nov 03 20:15:59.525965 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.525920    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3a7dfe6-8695-4589-84ad-dc8b844a437f" containerName="cleanup"
Nov 03 20:15:59.525965 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.525944    4421 cpu_manager.go:414] "containerMap: removing container" podUID="c3a7dfe6-8695-4589-84ad-dc8b844a437f" containerName="cleanup"
Nov 03 20:15:59.525965 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.525950    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3a7dfe6-8695-4589-84ad-dc8b844a437f" containerName="cleanup"
Nov 03 20:15:59.525965 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.525962    4421 cpu_manager.go:414] "containerMap: removing container" podUID="3058d8f4-76e2-4dd7-8b44-04c90b48c722" containerName="payload-processing"
Nov 03 20:15:59.526085 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.525972    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3058d8f4-76e2-4dd7-8b44-04c90b48c722" containerName="payload-processing"
Nov 03 20:15:59.526085 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.525993    4421 cpu_manager.go:414] "containerMap: removing container" podUID="c3a7dfe6-8695-4589-84ad-dc8b844a437f" containerName="cleanup"
Nov 03 20:15:59.526085 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.525998    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3a7dfe6-8695-4589-84ad-dc8b844a437f" containerName="cleanup"
Nov 03 20:15:59.526175 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.526107    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3a7dfe6-8695-4589-84ad-dc8b844a437f" containerName="cleanup"
Nov 03 20:15:59.526175 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.526114    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="3058d8f4-76e2-4dd7-8b44-04c90b48c722" containerName="payload-processing"
Nov 03 20:15:59.526175 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.526120    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3a7dfe6-8695-4589-84ad-dc8b844a437f" containerName="cleanup"
Nov 03 20:15:59.529152 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.529118    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.545945 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.545920    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"]
Nov 03 20:15:59.546642 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.546592    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3a7dfe6-8695-4589-84ad-dc8b844a437f" containerName="cleanup"
Nov 03 20:15:59.550229 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.550204    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"]
Nov 03 20:15:59.550339 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.550315    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.566587 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.566531    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"]
Nov 03 20:15:59.623239 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.623205    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.623469 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.623334    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv26v\" (UniqueName: \"kubernetes.io/projected/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-kube-api-access-cv26v\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.623469 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.623381    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bb10f7c-7a24-482b-961e-966a83c8139d-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.623469 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.623421    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.623469 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.623453    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.623705 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.623483    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.623705 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.623513    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.623705 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.623609    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.623705 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.623638    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.623705 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.623663    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.623705 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.623680    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzp5h\" (UniqueName: \"kubernetes.io/projected/7bb10f7c-7a24-482b-961e-966a83c8139d-kube-api-access-kzp5h\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.623956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.623761    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.623956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.623799    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.623956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.623844    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.623956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.623872    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.623956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.623898    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.623956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.623912    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.624140 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.623959    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.725073 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.725043    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv26v\" (UniqueName: \"kubernetes.io/projected/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-kube-api-access-cv26v\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.725283 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.725093    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bb10f7c-7a24-482b-961e-966a83c8139d-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.725283 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.725136    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.725283 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.725156    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.725283 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.725181    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.725283 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.725209    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.725558 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.725289    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.725558 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.725325    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.725558 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.725372    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.725558 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.725400    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzp5h\" (UniqueName: \"kubernetes.io/projected/7bb10f7c-7a24-482b-961e-966a83c8139d-kube-api-access-kzp5h\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.725558 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.725434    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.725558 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.725476    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.725558 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.725518    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.725558 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.725541    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.726026 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.725580    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.726026 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.725622    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.726026 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.725650    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.726026 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.725677    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.726026 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.725696    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.726026 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.725720    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.726026 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.725744    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.726026 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.726024    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.726427 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.726108    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bb10f7c-7a24-482b-961e-966a83c8139d-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.726427 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.726263    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.726427 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.726289    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.726627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.726587    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.726706 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.726536    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.726790 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.726766    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.728564 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.728523    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.728726 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.728648    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.728799 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.728732    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.729085 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.729069    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.738106 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.738080    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.738471 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.738453    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv26v\" (UniqueName: \"kubernetes.io/projected/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-kube-api-access-cv26v\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.738471 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.738463    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.743099 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.743079    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzp5h\" (UniqueName: \"kubernetes.io/projected/7bb10f7c-7a24-482b-961e-966a83c8139d-kube-api-access-kzp5h\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:15:59.845696 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.844944    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:15:59.886655 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:15:59.886354    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:16:00.166203 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:00.166175    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"]
Nov 03 20:16:00.167252 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:16:00.167217    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f6f3683_cb02_4dd2_aed6_a919a84c2e4c.slice/crio-079618efe6c4c882a28e2b5fb271f831b0aa49d9e09df021162b6bc3c3241a7c WatchSource:0}: Error finding container 079618efe6c4c882a28e2b5fb271f831b0aa49d9e09df021162b6bc3c3241a7c: Status 404 returned error can't find the container with id 079618efe6c4c882a28e2b5fb271f831b0aa49d9e09df021162b6bc3c3241a7c
Nov 03 20:16:00.169527 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:00.169488    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:00.169659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:00.169563    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:00.169659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:00.169618    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:00.189885 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:00.189865    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"]
Nov 03 20:16:00.200860 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:16:00.200833    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb10f7c_7a24_482b_961e_966a83c8139d.slice/crio-2ae5f460ef58ad83e7d88ad2398d50851cff6f2140fffaf9f3f3bb3c9320f298 WatchSource:0}: Error finding container 2ae5f460ef58ad83e7d88ad2398d50851cff6f2140fffaf9f3f3bb3c9320f298: Status 404 returned error can't find the container with id 2ae5f460ef58ad83e7d88ad2398d50851cff6f2140fffaf9f3f3bb3c9320f298
Nov 03 20:16:00.203208 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:00.203172    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:00.203278 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:00.203257    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:00.203314 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:00.203297    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:00.474701 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:00.474667    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb" event={"ID":"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c","Type":"ContainerStarted","Data":"49d100bfcb4c65564d1f8d4e077dcbac06ab79aa87b82ae2fa63d75be59ed94a"}
Nov 03 20:16:00.474910 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:00.474726    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb" event={"ID":"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c","Type":"ContainerStarted","Data":"079618efe6c4c882a28e2b5fb271f831b0aa49d9e09df021162b6bc3c3241a7c"}
Nov 03 20:16:00.476646 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:00.476617    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5" event={"ID":"7bb10f7c-7a24-482b-961e-966a83c8139d","Type":"ContainerStarted","Data":"cc649c45aff40c543c3d7cb57f11823f915340bf656528282609bb229d9c3af0"}
Nov 03 20:16:00.476772 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:00.476668    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5" event={"ID":"7bb10f7c-7a24-482b-961e-966a83c8139d","Type":"ContainerStarted","Data":"2ae5f460ef58ad83e7d88ad2398d50851cff6f2140fffaf9f3f3bb3c9320f298"}
Nov 03 20:16:00.502994 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:00.499579    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb" podStartSLOduration=1.499561685 podStartE2EDuration="1.499561685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:00.497777512 +0000 UTC m=+890.630905975" watchObservedRunningTime="2026-11-03 20:16:00.499561685 +0000 UTC m=+890.632690148"
Nov 03 20:16:00.524441 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:00.524375    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5" podStartSLOduration=1.524356893 podStartE2EDuration="1.524356893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:00.521129699 +0000 UTC m=+890.654258198" watchObservedRunningTime="2026-11-03 20:16:00.524356893 +0000 UTC m=+890.657485358"
Nov 03 20:16:00.868682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:00.864989    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:16:00.887068 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:00.886977    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:16:01.888842 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:01.888807    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:16:01.889400 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:01.889369    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5" podUID="7bb10f7c-7a24-482b-961e-966a83c8139d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 20:16:01.889878 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:01.889856    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:16:01.890052 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:01.890029    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb" podUID="8f6f3683-cb02-4dd2-aed6-a919a84c2e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 20:16:02.856996 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:02.856958    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:16:02.857321 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:02.857273    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb" podUID="8f6f3683-cb02-4dd2-aed6-a919a84c2e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 20:16:02.887287 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:02.887254    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:16:02.887579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:02.887543    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5" podUID="7bb10f7c-7a24-482b-961e-966a83c8139d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 20:16:03.858041 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:03.858001    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:16:03.858695 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:03.858659    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb" podUID="8f6f3683-cb02-4dd2-aed6-a919a84c2e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 20:16:03.889621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:03.888969    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:16:03.889621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:03.889031    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5" podUID="7bb10f7c-7a24-482b-961e-966a83c8139d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 20:16:04.860625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:04.860039    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:16:04.860625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:04.860116    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb" podUID="8f6f3683-cb02-4dd2-aed6-a919a84c2e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 20:16:04.890650 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:04.887961    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:16:04.890650 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:04.888035    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5" podUID="7bb10f7c-7a24-482b-961e-966a83c8139d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 20:16:05.862973 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:05.858981    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:16:05.862973 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:05.859037    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb" podUID="8f6f3683-cb02-4dd2-aed6-a919a84c2e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 20:16:05.893623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:05.890171    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:16:05.893623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:05.890246    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5" podUID="7bb10f7c-7a24-482b-961e-966a83c8139d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 20:16:06.852630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:06.851690    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:16:06.852630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:06.851749    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb" podUID="8f6f3683-cb02-4dd2-aed6-a919a84c2e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 20:16:06.888651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:06.887997    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:16:06.888651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:06.888060    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5" podUID="7bb10f7c-7a24-482b-961e-966a83c8139d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 20:16:07.058622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:07.058510    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:07.340623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:07.337961    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:07.854162 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:07.854121    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 20:16:07.854506 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:07.854454    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb" podUID="8f6f3683-cb02-4dd2-aed6-a919a84c2e4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 20:16:07.889109 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:07.889075    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 20:16:07.889725 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:07.889677    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5" podUID="7bb10f7c-7a24-482b-961e-966a83c8139d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 20:16:08.072730 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:08.072684    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:16:08.073800 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:08.073777    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:16:08.395621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:08.363472    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:16:08.395621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:08.364212    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:16:08.471624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:08.467012    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:16:08.497277 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:08.497245    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:16:08.521247 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:08.521108    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z" podUID="34c80431-a853-4012-b239-ea4659c70bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:09.044625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:08.999796    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:09.146669 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:09.146627    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 03 20:16:09.357878 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:09.357797    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:16:09.655043 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:09.654944    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:16:09.656496 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:09.656470    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:16:09.658094 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:09.658069    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:16:09.771316 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:09.771286    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:16:09.772538 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:09.772515    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:16:09.774131 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:09.774112    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:16:13.490741 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.490712    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"]
Nov 03 20:16:13.496621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.496583    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.516152 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.516130    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"]
Nov 03 20:16:13.554339 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.554308    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.554486 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.554372    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.554486 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.554445    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.554565 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.554511    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpwfn\" (UniqueName: \"kubernetes.io/projected/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-kube-api-access-kpwfn\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.554626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.554581    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.554733 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.554717    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.554798 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.554782    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.554835 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.554809    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.554870 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.554859    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.655744 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.655699    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.655932 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.655775    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.655932 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.655812    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.655932 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.655843    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpwfn\" (UniqueName: \"kubernetes.io/projected/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-kube-api-access-kpwfn\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.655932 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.655880    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.656151 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.655964    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.656151 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.656011    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.656151 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.656045    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.656151 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.656116    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.656151 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.656131    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.656483 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.656456    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.656863 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.656834    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.657190 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.657163    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.657308 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.657244    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.658953 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.658932    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.659657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.659636    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.669498 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.669475    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpwfn\" (UniqueName: \"kubernetes.io/projected/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-kube-api-access-kpwfn\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.669692 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.669667    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:13.823104 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:13.823028    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:14.137933 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.137909    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"]
Nov 03 20:16:14.139798 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:16:14.139765    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08eb9bd3_5d1e_4284_bd0e_e3fb8f10adfc.slice/crio-066ab2105adc1581c6a3816bc5066fca3916efe6ce3a1be44302db7ebd201637 WatchSource:0}: Error finding container 066ab2105adc1581c6a3816bc5066fca3916efe6ce3a1be44302db7ebd201637: Status 404 returned error can't find the container with id 066ab2105adc1581c6a3816bc5066fca3916efe6ce3a1be44302db7ebd201637
Nov 03 20:16:14.143353 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.143310    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:14.143462 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.143416    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:14.143517 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.143463    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:14.160833 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.160806    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm" event={"ID":"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc","Type":"ContainerStarted","Data":"066ab2105adc1581c6a3816bc5066fca3916efe6ce3a1be44302db7ebd201637"}
Nov 03 20:16:14.575539 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.575504    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"]
Nov 03 20:16:14.580796 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.580769    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:14.608624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.605810    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"]
Nov 03 20:16:14.795970 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.795925    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:14.796167 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.795983    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e103d59c-e7b4-4713-a826-d3a8834f947a-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:14.796167 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.796016    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:14.796167 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.796148    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:14.796324 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.796177    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:14.796324 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.796217    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:14.796324 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.796258    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:14.796324 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.796291    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvb9t\" (UniqueName: \"kubernetes.io/projected/e103d59c-e7b4-4713-a826-d3a8834f947a-kube-api-access-zvb9t\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:14.796527 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.796356    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:14.897422 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.897320    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:14.897422 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.897376    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:14.897422 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.897402    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:14.897422 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.897431    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:14.897853 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.897453    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvb9t\" (UniqueName: \"kubernetes.io/projected/e103d59c-e7b4-4713-a826-d3a8834f947a-kube-api-access-zvb9t\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:14.897853 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.897512    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:14.897853 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.897670    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:14.897853 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.897705    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e103d59c-e7b4-4713-a826-d3a8834f947a-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:14.897853 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.897733    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:14.897853 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.897800    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:14.898075 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.897901    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:14.898075 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.897959    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:14.898140 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.898072    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:14.898384 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.898364    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e103d59c-e7b4-4713-a826-d3a8834f947a-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:14.900126 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.900100    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:14.900321 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.900303    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:14.909962 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.909932    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:14.910538 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:14.910517    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvb9t\" (UniqueName: \"kubernetes.io/projected/e103d59c-e7b4-4713-a826-d3a8834f947a-kube-api-access-zvb9t\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:15.184706 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:15.183713    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm" event={"ID":"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc","Type":"ContainerStarted","Data":"bde8000b904e5f5f581abdb127e2a365a8b36201d2e4dfc6815a921bb8d05cfe"}
Nov 03 20:16:15.251196 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:15.249495    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:15.283455 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:15.283069    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm" podStartSLOduration=2.283044048 podStartE2EDuration="2.283044048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:15.23204083 +0000 UTC m=+905.365169293" watchObservedRunningTime="2026-11-03 20:16:15.283044048 +0000 UTC m=+905.416172511"
Nov 03 20:16:15.597347 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:15.597310    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"]
Nov 03 20:16:15.598418 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:16:15.598392    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode103d59c_e7b4_4713_a826_d3a8834f947a.slice/crio-64f003fb24bb2c58be60a837f4b8ca01dd16c5a9488eb7202561ab6be3f41fe6 WatchSource:0}: Error finding container 64f003fb24bb2c58be60a837f4b8ca01dd16c5a9488eb7202561ab6be3f41fe6: Status 404 returned error can't find the container with id 64f003fb24bb2c58be60a837f4b8ca01dd16c5a9488eb7202561ab6be3f41fe6
Nov 03 20:16:15.600891 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:15.600859    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:15.600966 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:15.600940    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:15.601005 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:15.600977    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:15.825269 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:15.825132    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:15.827324 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:15.826355    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:16:15.827324 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:15.826417    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm" podUID="08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 20:16:16.191125 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:16.191090    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv" event={"ID":"e103d59c-e7b4-4713-a826-d3a8834f947a","Type":"ContainerStarted","Data":"aedf83c4e6a2ef5dbdb3cd08339a54da45bd4bd0fa18a46028ee37acfec4b82b"}
Nov 03 20:16:16.191308 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:16.191157    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv" event={"ID":"e103d59c-e7b4-4713-a826-d3a8834f947a","Type":"ContainerStarted","Data":"64f003fb24bb2c58be60a837f4b8ca01dd16c5a9488eb7202561ab6be3f41fe6"}
Nov 03 20:16:16.259368 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:16.257659    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:16.259368 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:16.258189    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv" podStartSLOduration=2.258172112 podStartE2EDuration="2.258172112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:16.257410734 +0000 UTC m=+906.390539197" watchObservedRunningTime="2026-11-03 20:16:16.258172112 +0000 UTC m=+906.391300575"
Nov 03 20:16:16.265623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:16.261708    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:16:16.265623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:16.261769    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv" podUID="e103d59c-e7b4-4713-a826-d3a8834f947a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:16:16.835775 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:16.835740    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:16:16.837703 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:16.837669    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm" podUID="08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 20:16:17.253014 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:17.252972    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:16:17.253292 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:17.253263    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv" podUID="e103d59c-e7b4-4713-a826-d3a8834f947a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:16:17.824348 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:17.824320    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:16:17.824514 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:17.824367    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm" podUID="08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 20:16:18.261624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:18.261033    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:16:18.261624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:18.261102    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv" podUID="e103d59c-e7b4-4713-a826-d3a8834f947a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:16:18.823473 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:18.823440    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:16:18.823686 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:18.823487    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm" podUID="08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 20:16:19.282647 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:19.273660    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:16:19.282647 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:19.273723    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv" podUID="e103d59c-e7b4-4713-a826-d3a8834f947a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:16:19.629793 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:16:19.629689    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:16:19.823884 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:19.823848    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:16:19.824053 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:19.823900    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm" podUID="08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 20:16:20.251982 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:20.251947    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:16:20.252243 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:20.252218    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv" podUID="e103d59c-e7b4-4713-a826-d3a8834f947a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:16:20.824231 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:20.824200    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 20:16:20.824664 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:20.824253    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm" podUID="08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 20:16:21.258625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:21.252161    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:16:21.258625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:21.252218    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv" podUID="e103d59c-e7b4-4713-a826-d3a8834f947a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:16:21.856081 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:21.854952    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:21.856081 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:21.855813    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:21.856956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:21.856920    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:16:22.250227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:22.250198    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 20:16:22.250374 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:22.250248    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv" podUID="e103d59c-e7b4-4713-a826-d3a8834f947a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 20:16:23.581690 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:23.581660    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:23.582257 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:23.582122    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:23.583361 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:23.583342    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 20:16:24.799527 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.799488    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"]
Nov 03 20:16:24.804854 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.804824    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:24.828666 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.828635    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"]
Nov 03 20:16:24.881978 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.881946    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:24.882166 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.881994    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j87d\" (UniqueName: \"kubernetes.io/projected/c9adc7d2-3c5b-42e1-b790-20db31c15593-kube-api-access-7j87d\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:24.882166 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.882017    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:24.882343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.882192    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:24.882402 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.882367    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:24.882453 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.882424    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9adc7d2-3c5b-42e1-b790-20db31c15593-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:24.882531 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.882515    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:24.882579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.882552    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:24.882678 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.882662    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:24.983573 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.983541    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:24.983782 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.983658    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:24.983782 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.983684    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9adc7d2-3c5b-42e1-b790-20db31c15593-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:24.983782 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.983735    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:24.983782 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.983767    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:24.984016 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.983817    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:24.984016 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.983972    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:24.984132 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.984028    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j87d\" (UniqueName: \"kubernetes.io/projected/c9adc7d2-3c5b-42e1-b790-20db31c15593-kube-api-access-7j87d\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:24.984132 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.984059    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:24.984850 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.984576    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:24.986949 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.985120    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:24.986949 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.985286    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:24.986949 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.985518    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:24.986949 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.985924    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9adc7d2-3c5b-42e1-b790-20db31c15593-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:24.986949 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.986685    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:24.988855 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:24.988108    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:25.001165 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:25.001128    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:25.001408 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:25.001364    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j87d\" (UniqueName: \"kubernetes.io/projected/c9adc7d2-3c5b-42e1-b790-20db31c15593-kube-api-access-7j87d\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:25.123755 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:25.123437    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:25.482730 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:25.482642    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:25.482983 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:25.482960    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:25.483139 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:25.483113    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:25.499901 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:25.498347    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"]
Nov 03 20:16:26.299114 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:26.299068    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d" event={"ID":"c9adc7d2-3c5b-42e1-b790-20db31c15593","Type":"ContainerStarted","Data":"8b5c105b65710bcd692af2980de1b6e91d53e47ae87cbd94742b3a21ba0d81cd"}
Nov 03 20:16:26.318930 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:26.318891    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d" event={"ID":"c9adc7d2-3c5b-42e1-b790-20db31c15593","Type":"ContainerStarted","Data":"6d34843ec925038aa75fdd494bd40c0ca4be42d67c37b4935cd05d4cd958f4cc"}
Nov 03 20:16:26.372626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:26.372092    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d" podStartSLOduration=2.372068158 podStartE2EDuration="2.372068158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:26.341167564 +0000 UTC m=+916.474296027" watchObservedRunningTime="2026-11-03 20:16:26.372068158 +0000 UTC m=+916.505196625"
Nov 03 20:16:26.996915 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:26.995693    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:27.128625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:27.128455    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:27.132621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:27.129468    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:16:27.132621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:27.129534    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d" podUID="c9adc7d2-3c5b-42e1-b790-20db31c15593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 20:16:27.170625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:27.155564    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:27.395683 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:27.389683    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:27.512725 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:27.512674    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-6bhmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:16:27.513048 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:27.513016    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr" podUID="9d6e2ee7-7d48-4879-b6fa-f303b5d24697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:27.797862 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:27.797813    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:27.944621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:27.944030    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-cd5ll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:16:27.944621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:27.944118    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll" podUID="b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:28.005375 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:28.005021    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 20:16:28.006389 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:28.006352    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:16:28.137509 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:28.136998    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:16:28.137509 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:28.137058    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d" podUID="c9adc7d2-3c5b-42e1-b790-20db31c15593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 20:16:28.198634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:28.172786    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:28.198634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:28.173485    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:16:28.436697 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:28.415396    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:28.436697 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:28.415937    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:16:28.489551 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:28.487562    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:28.682222 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:28.682187    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:16:28.682524 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:28.682496    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw" podUID="317e6b68-68f8-4220-a3ef-2f0999a19ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:28.822713 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:28.819697    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:28.842624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:28.823085    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:16:29.057749 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:29.057699    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:29.174320 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:29.174227    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 03 20:16:29.182193 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:29.182161    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:16:29.182452 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:29.182425    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d" podUID="c9adc7d2-3c5b-42e1-b790-20db31c15593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 20:16:29.416205 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:29.416149    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:16:29.493753 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:29.493702    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:16:29.500410 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:29.498493    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:16:29.793822 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:29.793700    4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:29.827622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:29.826975    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:16:30.154381 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:30.139699    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 20:16:30.154381 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:30.140946    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:16:30.154381 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:30.140983    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d" podUID="c9adc7d2-3c5b-42e1-b790-20db31c15593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 20:16:30.212739 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:30.212689    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:30.775489 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:30.775451    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:16:30.822635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:30.803243    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:16:30.888326 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:30.827506    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:30.909706 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:30.904042    4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:31.173289 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:31.173184    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:31.193935 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:31.193887    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:16:31.240165 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:31.240125    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d" podUID="c9adc7d2-3c5b-42e1-b790-20db31c15593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 20:16:31.353308 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:31.352947    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 03 20:16:31.384624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:31.364659    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj" podUID="40f4d063-2060-49f8-8b17-13526284ed89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:31.658633 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:31.652176    4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:31.669213 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:31.669175    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:31.859190 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:31.859147    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:31.862014 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:31.861977    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:32.144785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:32.144734    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:16:32.145087 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:32.145058    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d" podUID="c9adc7d2-3c5b-42e1-b790-20db31c15593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 20:16:32.183024 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:32.182699    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:32.356738 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:32.356686    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:32.531241 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:32.531211    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:16:32.533213 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:32.533189    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:16:32.609778 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:32.608849    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:16:32.729245 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:32.726238    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:16:33.125622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:33.125008    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:16:33.125622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:33.125087    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d" podUID="c9adc7d2-3c5b-42e1-b790-20db31c15593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 20:16:34.134322 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:34.134273    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:16:34.134873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:34.134362    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d" podUID="c9adc7d2-3c5b-42e1-b790-20db31c15593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 20:16:35.124481 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:35.124425    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:16:35.124828 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:35.124511    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d" podUID="c9adc7d2-3c5b-42e1-b790-20db31c15593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 20:16:36.129012 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:36.128978    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 20:16:36.129629 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:36.129561    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d" podUID="c9adc7d2-3c5b-42e1-b790-20db31c15593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 20:16:37.267428 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:37.267393    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:37.267848 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:37.267832    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:37.268780 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:37.268760    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 20:16:47.183465 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:47.183430    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"]
Nov 03 20:16:47.210875 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:47.210841    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:16:47.211061 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:47.210741    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"]
Nov 03 20:16:47.218483 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:47.218458    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b13a14\"/\"default-dockercfg-tvmvj\""
Nov 03 20:16:47.220910 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:47.220882    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b13a14\"/\"kube-root-ca.crt\""
Nov 03 20:16:47.221198 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:47.221175    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b13a14\"/\"distinct-2-w1-b13a14-kserve-self-signed-certs\""
Nov 03 20:16:47.221362 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:47.221348    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b13a14\"/\"openshift-service-ca.crt\""
Nov 03 20:16:47.348670 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:47.337714    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-tmp-dir\") pod \"distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r\" (UID: \"b7be6b1b-1739-448a-9ca7-27556b447c0f\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:16:47.348978 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:47.348944    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-dshm\") pod \"distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r\" (UID: \"b7be6b1b-1739-448a-9ca7-27556b447c0f\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:16:47.349127 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:47.349111    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-home\") pod \"distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r\" (UID: \"b7be6b1b-1739-448a-9ca7-27556b447c0f\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:16:47.349527 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:47.349502    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7be6b1b-1739-448a-9ca7-27556b447c0f-tls-certs\") pod \"distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r\" (UID: \"b7be6b1b-1739-448a-9ca7-27556b447c0f\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:16:47.365752 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:47.365705    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwxcg\" (UniqueName: \"kubernetes.io/projected/b7be6b1b-1739-448a-9ca7-27556b447c0f-kube-api-access-hwxcg\") pod \"distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r\" (UID: \"b7be6b1b-1739-448a-9ca7-27556b447c0f\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:16:47.366076 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:47.366044    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-model-cache\") pod \"distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r\" (UID: \"b7be6b1b-1739-448a-9ca7-27556b447c0f\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:16:47.467977 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:47.467933    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7be6b1b-1739-448a-9ca7-27556b447c0f-tls-certs\") pod \"distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r\" (UID: \"b7be6b1b-1739-448a-9ca7-27556b447c0f\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:16:47.468196 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:47.467982    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwxcg\" (UniqueName: \"kubernetes.io/projected/b7be6b1b-1739-448a-9ca7-27556b447c0f-kube-api-access-hwxcg\") pod \"distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r\" (UID: \"b7be6b1b-1739-448a-9ca7-27556b447c0f\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:16:47.468196 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:47.468016    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-model-cache\") pod \"distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r\" (UID: \"b7be6b1b-1739-448a-9ca7-27556b447c0f\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:16:47.468196 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:47.468073    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-tmp-dir\") pod \"distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r\" (UID: \"b7be6b1b-1739-448a-9ca7-27556b447c0f\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:16:47.468196 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:47.468099    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-dshm\") pod \"distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r\" (UID: \"b7be6b1b-1739-448a-9ca7-27556b447c0f\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:16:47.468196 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:47.468128    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-home\") pod \"distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r\" (UID: \"b7be6b1b-1739-448a-9ca7-27556b447c0f\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:16:47.468546 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:47.468518    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-model-cache\") pod \"distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r\" (UID: \"b7be6b1b-1739-448a-9ca7-27556b447c0f\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:16:47.468678 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:47.468562    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-home\") pod \"distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r\" (UID: \"b7be6b1b-1739-448a-9ca7-27556b447c0f\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:16:47.468986 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:47.468958    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-tmp-dir\") pod \"distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r\" (UID: \"b7be6b1b-1739-448a-9ca7-27556b447c0f\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:16:47.471184 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:47.471161    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-dshm\") pod \"distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r\" (UID: \"b7be6b1b-1739-448a-9ca7-27556b447c0f\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:16:47.471346 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:47.471328    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7be6b1b-1739-448a-9ca7-27556b447c0f-tls-certs\") pod \"distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r\" (UID: \"b7be6b1b-1739-448a-9ca7-27556b447c0f\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:16:47.483468 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:47.483441    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwxcg\" (UniqueName: \"kubernetes.io/projected/b7be6b1b-1739-448a-9ca7-27556b447c0f-kube-api-access-hwxcg\") pod \"distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r\" (UID: \"b7be6b1b-1739-448a-9ca7-27556b447c0f\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:16:47.536523 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:47.536492    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:16:47.830054 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:47.830024    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"]
Nov 03 20:16:48.573470 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:48.573426    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" event={"ID":"b7be6b1b-1739-448a-9ca7-27556b447c0f","Type":"ContainerStarted","Data":"af60b8e04fc5efed6264f40614db441743fa2556a13e5562846220f344eebd0a"}
Nov 03 20:16:48.573916 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:48.573493    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" event={"ID":"b7be6b1b-1739-448a-9ca7-27556b447c0f","Type":"ContainerStarted","Data":"69ec73d098771c1c182352c54047a009d1125141772c2c865eed31291cce8d03"}
Nov 03 20:16:48.603623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:48.602532    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podStartSLOduration=1.602513816 podStartE2EDuration="1.602513816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:48.602145927 +0000 UTC m=+938.735274390" watchObservedRunningTime="2026-11-03 20:16:48.602513816 +0000 UTC m=+938.735642279"
Nov 03 20:16:49.543773 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.543685    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:16:49.696947 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.696911    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"]
Nov 03 20:16:49.715941 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.715912    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:49.802721 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.800681    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:49.802721 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.800758    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8sfl\" (UniqueName: \"kubernetes.io/projected/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-kube-api-access-g8sfl\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:49.802721 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.800822    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:49.802721 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.800852    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:49.802721 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.800895    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:49.802721 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.800929    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:49.802721 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.800975    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:49.802721 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.801071    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:49.802721 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.801098    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:49.835105 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.835068    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"]
Nov 03 20:16:49.856627 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:16:49.849458    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:16:49.909635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.907493    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:49.909635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.908643    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:49.909635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.908678    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:49.909635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.908785    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:49.909635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.908841    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8sfl\" (UniqueName: \"kubernetes.io/projected/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-kube-api-access-g8sfl\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:49.909635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.908916    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:49.909635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.908949    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:49.909635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.908994    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:49.909635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.909029    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:49.920699 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.920341    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:49.921058 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.921030    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:49.924626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.921298    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:49.924626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.921558    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:49.924626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.921827    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:49.974905 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:49.974795    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:50.031925 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:50.031868    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:50.042210 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:50.042146    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:50.063139 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:50.063061    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:50.149354 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:50.149274    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8sfl\" (UniqueName: \"kubernetes.io/projected/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-kube-api-access-g8sfl\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:50.177626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:50.177120    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 03 20:16:50.181984 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:50.181665    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 20:16:50.350936 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:50.350837    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:50.724633 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:50.718712    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:51.001679 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:50.986042    4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:51.002133 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:51.002092    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:16:51.128622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:51.110774    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:51.128622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:51.112336    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:16:51.184625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:51.183686    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 20:16:51.184625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:51.184481    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:16:51.286674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:51.185641    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 03 20:16:51.286674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:51.185977    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:16:51.903826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:51.903780    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:51.936059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:51.905507    4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": context deadline exceeded"
Nov 03 20:16:51.956113 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:51.938358    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:16:52.142290 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:52.126005    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:16:52.157169 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:52.149188    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:16:52.170786 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:52.170716    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:52.283042 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:52.191742    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:52.284227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:52.284186    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:52.808624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:52.804985    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:52.808624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:52.807236    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:16:52.908186 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:52.908125    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:16:53.043622 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:16:53.030431    4421 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.009s"
Nov 03 20:16:53.043622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:53.030478    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"]
Nov 03 20:16:53.043622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:53.030637    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:16:53.044055 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:53.044024    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:16:53.063806 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:53.063707    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn" podUID="d31c0deb-63e5-42a2-a34c-90499e741b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:53.151625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:53.125345    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4" event={"ID":"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6","Type":"ContainerStarted","Data":"11437d2ec6d9f8b10414028bd1525a6d17ceccf661129c117a55ce76c5028e65"}
Nov 03 20:16:53.171838 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:53.154045    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:53.171838 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:53.158039    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:53.350682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:53.322029    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 20:16:53.350682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:53.322946    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 03 20:16:53.689676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:53.684935    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:16:53.689676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:53.684999    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm" podUID="08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:53.779539 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:53.779506    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:16:53.780374 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:53.779812    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z" podUID="34c80431-a853-4012-b239-ea4659c70bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:53.806011 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:53.805970    4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded"
Nov 03 20:16:53.810976 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:53.808709    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:16:53.811205 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:53.811176    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d" podUID="c9adc7d2-3c5b-42e1-b790-20db31c15593" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:53.911741 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:53.911692    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:16:53.993897 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:53.993860    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-w9chd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:16:53.994161 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:53.994132    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd" podUID="657622a4-6298-4496-b2c3-edce95190d04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:54.261125 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:54.260736    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:54.268337 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:54.267692    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:16:54.268337 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:54.267749    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:16:54.333835 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:54.333793    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:16:54.632698 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:54.632377    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:16:54.704733 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:54.687283    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:16:54.758024 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:54.757999    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:16:55.670740 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:55.670698    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4" event={"ID":"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6","Type":"ContainerStarted","Data":"64b6998a90bc28dcd29cac55c072d91189e307ab590b427480be9360ad220d7c"}
Nov 03 20:16:55.697772 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:55.697721    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4" podStartSLOduration=6.697708051 podStartE2EDuration="6.697708051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:55.694184992 +0000 UTC m=+945.827313458" watchObservedRunningTime="2026-11-03 20:16:55.697708051 +0000 UTC m=+945.830836513"
Nov 03 20:16:56.357623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:56.354634    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:16:56.371650 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:56.371529    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 20:16:56.371918 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:56.371871    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4" podUID="6b34f6ef-a056-4dbb-9e6f-d61c78d770b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 20:16:57.360622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:57.360251    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 20:16:57.365712 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:57.365661    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4" podUID="6b34f6ef-a056-4dbb-9e6f-d61c78d770b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 20:16:57.536898 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:57.536865    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:16:57.576908 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:57.576878    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:16:57.694778 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:57.694753    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:16:58.362433 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:58.362394    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 20:16:58.363194 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:58.363161    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4" podUID="6b34f6ef-a056-4dbb-9e6f-d61c78d770b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 20:16:59.356002 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:59.355967    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 20:16:59.356291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:16:59.356266    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4" podUID="6b34f6ef-a056-4dbb-9e6f-d61c78d770b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 20:17:00.358624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:00.354936    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 20:17:00.358624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:00.354991    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4" podUID="6b34f6ef-a056-4dbb-9e6f-d61c78d770b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 20:17:01.351756 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:01.351720    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 20:17:01.351931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:01.351789    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4" podUID="6b34f6ef-a056-4dbb-9e6f-d61c78d770b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 20:17:02.386166 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:02.386136    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:17:02.386770 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:02.386569    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:17:02.387662 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:02.387640    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:17:06.875846 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:06.875815    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-db5557-ffc78867d-sggqh"]
Nov 03 20:17:06.880447 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:06.880422    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-db5557-ffc78867d-sggqh"
Nov 03 20:17:06.890971 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:06.890937    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-db5557-ffc78867d-sggqh"]
Nov 03 20:17:06.891915 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:06.891876    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-db5557-dockercfg-l4h4b\""
Nov 03 20:17:06.907868 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:06.907842    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg2qw\" (UniqueName: \"kubernetes.io/projected/c032b61d-02e3-4c2d-b7e7-2768ec0c62cf-kube-api-access-wg2qw\") pod \"payload-processing-e2e-api-a-db5557-ffc78867d-sggqh\" (UID: \"c032b61d-02e3-4c2d-b7e7-2768ec0c62cf\") " pod="openshift-ingress/payload-processing-e2e-api-a-db5557-ffc78867d-sggqh"
Nov 03 20:17:06.908053 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:06.908032    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c032b61d-02e3-4c2d-b7e7-2768ec0c62cf-plugins-config-volume\") pod \"payload-processing-e2e-api-a-db5557-ffc78867d-sggqh\" (UID: \"c032b61d-02e3-4c2d-b7e7-2768ec0c62cf\") " pod="openshift-ingress/payload-processing-e2e-api-a-db5557-ffc78867d-sggqh"
Nov 03 20:17:06.972632 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:06.955582    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-db5557\""
Nov 03 20:17:07.017375 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:07.017334    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c032b61d-02e3-4c2d-b7e7-2768ec0c62cf-plugins-config-volume\") pod \"payload-processing-e2e-api-a-db5557-ffc78867d-sggqh\" (UID: \"c032b61d-02e3-4c2d-b7e7-2768ec0c62cf\") " pod="openshift-ingress/payload-processing-e2e-api-a-db5557-ffc78867d-sggqh"
Nov 03 20:17:07.033006 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:07.032961    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg2qw\" (UniqueName: \"kubernetes.io/projected/c032b61d-02e3-4c2d-b7e7-2768ec0c62cf-kube-api-access-wg2qw\") pod \"payload-processing-e2e-api-a-db5557-ffc78867d-sggqh\" (UID: \"c032b61d-02e3-4c2d-b7e7-2768ec0c62cf\") " pod="openshift-ingress/payload-processing-e2e-api-a-db5557-ffc78867d-sggqh"
Nov 03 20:17:07.040869 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:07.018467    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c032b61d-02e3-4c2d-b7e7-2768ec0c62cf-plugins-config-volume\") pod \"payload-processing-e2e-api-a-db5557-ffc78867d-sggqh\" (UID: \"c032b61d-02e3-4c2d-b7e7-2768ec0c62cf\") " pod="openshift-ingress/payload-processing-e2e-api-a-db5557-ffc78867d-sggqh"
Nov 03 20:17:07.063561 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:07.063523    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg2qw\" (UniqueName: \"kubernetes.io/projected/c032b61d-02e3-4c2d-b7e7-2768ec0c62cf-kube-api-access-wg2qw\") pod \"payload-processing-e2e-api-a-db5557-ffc78867d-sggqh\" (UID: \"c032b61d-02e3-4c2d-b7e7-2768ec0c62cf\") " pod="openshift-ingress/payload-processing-e2e-api-a-db5557-ffc78867d-sggqh"
Nov 03 20:17:07.206071 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:07.206024    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-db5557-ffc78867d-sggqh"
Nov 03 20:17:07.496709 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:07.496679    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-db5557-ffc78867d-sggqh"]
Nov 03 20:17:07.497586 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:17:07.497554    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc032b61d_02e3_4c2d_b7e7_2768ec0c62cf.slice/crio-95d5c0cb4fdca2a0304fddab134032b3f6f31bf8d81f2639b4b525b7dd8958a1 WatchSource:0}: Error finding container 95d5c0cb4fdca2a0304fddab134032b3f6f31bf8d81f2639b4b525b7dd8958a1: Status 404 returned error can't find the container with id 95d5c0cb4fdca2a0304fddab134032b3f6f31bf8d81f2639b4b525b7dd8958a1
Nov 03 20:17:07.793923 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:07.793811    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-db5557-ffc78867d-sggqh" event={"ID":"c032b61d-02e3-4c2d-b7e7-2768ec0c62cf","Type":"ContainerStarted","Data":"f946f087823ccc5e293edc86f145d53bc6f6af507fbd99e1a909af709cb44f94"}
Nov 03 20:17:07.794100 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:07.793929    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-db5557-ffc78867d-sggqh"
Nov 03 20:17:07.794100 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:07.793957    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-db5557-ffc78867d-sggqh" event={"ID":"c032b61d-02e3-4c2d-b7e7-2768ec0c62cf","Type":"ContainerStarted","Data":"95d5c0cb4fdca2a0304fddab134032b3f6f31bf8d81f2639b4b525b7dd8958a1"}
Nov 03 20:17:07.818719 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:07.818583    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-db5557-ffc78867d-sggqh" podStartSLOduration=1.818569648 podStartE2EDuration="1.818569648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:07.815858337 +0000 UTC m=+957.948986799" watchObservedRunningTime="2026-11-03 20:17:07.818569648 +0000 UTC m=+957.951698132"
Nov 03 20:17:11.148625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:11.147708    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:11.296645 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:11.293222    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:11.342616 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:11.341701    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:11.390623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:11.386747    4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:11.702960 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:11.701889    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:11.774626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:11.773700    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:12.222688 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:12.130236    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:12.432111 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:12.237817    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:12.876636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:12.741831    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:17:12.876636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:12.328693    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:12.876636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:12.542037    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:13.033072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:13.033025    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:17:13.127407 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:17:13.127308    4421 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.053s"
Nov 03 20:17:13.127776 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:13.127758    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:17:13.129736 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:13.129716    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:17:13.129958 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:13.129935    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:17:13.133627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:13.130093    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:17:13.133847 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:13.133832    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:17:13.156821 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:13.156780    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:17:13.158187 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:13.158163    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:17:13.290627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:13.265100    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:17:13.566624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:13.565773    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:17:13.623248 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:13.623219    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:17:13.650636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:13.641058    4421 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:17:13.650923 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:13.650890    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4" podUID="830b16e0-459d-444c-b7da-c8d6d8576d4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:13.745765 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:13.745707    4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded"
Nov 03 20:17:13.758619 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:13.746019    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 20:17:13.794750 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:13.794708    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:13.879800 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:13.879707    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:17:13.889044 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:13.888999    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw" podUID="317e6b68-68f8-4220-a3ef-2f0999a19ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:14.059629 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:14.039167    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:17:14.293055 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:14.187791    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:14.312003 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:14.286409    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:14.572771 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:14.572679    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:14.840026 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:14.839922    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:14.966031 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:14.966000    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:17:15.023210 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:15.023177    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"]
Nov 03 20:17:15.040349 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:15.040305    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:15.048323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:15.048296    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:15.205141 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:15.205052    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"]
Nov 03 20:17:15.247559 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:15.247517    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-djgdm\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:15.247864 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:15.247835    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ba69b5-5aac-45b5-a324-1557e64456e8-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-djgdm\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:15.248056 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:15.248036    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-djgdm\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:15.248194 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:15.248178    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-djgdm\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:15.248330 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:15.248313    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r829h\" (UniqueName: \"kubernetes.io/projected/c6ba69b5-5aac-45b5-a324-1557e64456e8-kube-api-access-r829h\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-djgdm\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:15.248485 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:15.248469    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-djgdm\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:15.248638 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:15.248622    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-djgdm\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:15.248768 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:15.248753    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-djgdm\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:15.248907 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:15.248892    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-djgdm\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:15.379362 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:15.360738    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-djgdm\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:15.379362 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:15.360792    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ba69b5-5aac-45b5-a324-1557e64456e8-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-djgdm\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:15.379362 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:15.360849    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-djgdm\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:15.379362 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:15.360872    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-djgdm\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:15.379362 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:15.360914    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r829h\" (UniqueName: \"kubernetes.io/projected/c6ba69b5-5aac-45b5-a324-1557e64456e8-kube-api-access-r829h\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-djgdm\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:15.379362 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:15.360970    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-djgdm\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:15.379362 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:15.361008    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-djgdm\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:15.379362 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:15.361039    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-djgdm\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:15.379362 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:15.361088    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-djgdm\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:15.802681 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:15.728642    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-djgdm\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:15.844572 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:15.844530    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-djgdm\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:15.845746 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:15.845716    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-djgdm\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:15.869979 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:15.869946    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ba69b5-5aac-45b5-a324-1557e64456e8-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-djgdm\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:16.015989 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:16.015945    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-djgdm\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:16.045493 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:16.031234    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-djgdm\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:16.110346 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:16.104840    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:17:16.346807 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:16.346765    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-djgdm\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:16.512417 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:16.512374    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-djgdm\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:17.055085 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:17.055039    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r829h\" (UniqueName: \"kubernetes.io/projected/c6ba69b5-5aac-45b5-a324-1557e64456e8-kube-api-access-r829h\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-djgdm\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:17.107937 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:17.107905    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:17.172066 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:17.170734    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:17.834751 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:17.832521    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:17:18.129642 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:18.129615    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"]
Nov 03 20:17:18.130530 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:17:18.130493    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ba69b5_5aac_45b5_a324_1557e64456e8.slice/crio-e20f6378aa4c9a28382bbe14ea873750a84a8c2513d2d109c0697097494b39e8 WatchSource:0}: Error finding container e20f6378aa4c9a28382bbe14ea873750a84a8c2513d2d109c0697097494b39e8: Status 404 returned error can't find the container with id e20f6378aa4c9a28382bbe14ea873750a84a8c2513d2d109c0697097494b39e8
Nov 03 20:17:18.133319 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:18.133277    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:17:18.133416 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:18.133368    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:17:18.133459 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:18.133411    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:17:18.816831 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:18.816802    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-db5557-ffc78867d-sggqh"
Nov 03 20:17:19.006365 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:19.006329    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm" event={"ID":"c6ba69b5-5aac-45b5-a324-1557e64456e8","Type":"ContainerStarted","Data":"5ae143c271475e2f747bff31a1a6e1cbbe93c20a5f64673e2f34dd433a0016a8"}
Nov 03 20:17:19.006886 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:19.006392    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm" event={"ID":"c6ba69b5-5aac-45b5-a324-1557e64456e8","Type":"ContainerStarted","Data":"e20f6378aa4c9a28382bbe14ea873750a84a8c2513d2d109c0697097494b39e8"}
Nov 03 20:17:19.031929 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:19.031878    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm" podStartSLOduration=5.031864283 podStartE2EDuration="5.031864283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:19.029936679 +0000 UTC m=+969.163065168" watchObservedRunningTime="2026-11-03 20:17:19.031864283 +0000 UTC m=+969.164992745"
Nov 03 20:17:19.108547 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:19.108462    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:19.109873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:19.109842    4421 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 20:17:19.109991 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:19.109889    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm" podUID="c6ba69b5-5aac-45b5-a324-1557e64456e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 20:17:19.630303 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:17:19.630267    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:17:20.109213 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:20.109178    4421 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 20:17:20.109623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:20.109224    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm" podUID="c6ba69b5-5aac-45b5-a324-1557e64456e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 20:17:21.108886 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:21.108855    4421 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 20:17:21.109067 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:21.108913    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm" podUID="c6ba69b5-5aac-45b5-a324-1557e64456e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 20:17:22.109351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:22.109312    4421 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 20:17:22.109802 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:22.109371    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm" podUID="c6ba69b5-5aac-45b5-a324-1557e64456e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 20:17:23.108824 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:23.108788    4421 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 20:17:23.109034 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:23.108862    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm" podUID="c6ba69b5-5aac-45b5-a324-1557e64456e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 20:17:24.109090 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:24.109055    4421 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 20:17:24.109640 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:24.109125    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm" podUID="c6ba69b5-5aac-45b5-a324-1557e64456e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 20:17:25.109103 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:25.109071    4421 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 20:17:25.109648 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:25.109120    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm" podUID="c6ba69b5-5aac-45b5-a324-1557e64456e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 20:17:26.114106 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:26.114075    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:26.114528 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:26.114378    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:26.115077 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:26.115058    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 20:17:34.725708 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:34.725666    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"]
Nov 03 20:17:35.053018 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.052931    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"]
Nov 03 20:17:35.053947 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.053926    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.074064 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.063401    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:35.238694 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.238655    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtl8n\" (UniqueName: \"kubernetes.io/projected/d5b6a788-2a4f-4819-8943-5fe88488888c-kube-api-access-xtl8n\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.239108 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.239081    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.239255 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.239237    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b6a788-2a4f-4819-8943-5fe88488888c-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.239484 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.239466    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.239667 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.239650    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.239819 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.239801    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.240434 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.240398    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.240532 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.240462    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.240624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.240534    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.375848 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.369993    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.375848 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.370050    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b6a788-2a4f-4819-8943-5fe88488888c-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.375848 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.370176    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.375848 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.370213    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.375848 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.370271    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.375848 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.370328    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.375848 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.370375    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.375848 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.370444    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.375848 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.370496    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtl8n\" (UniqueName: \"kubernetes.io/projected/d5b6a788-2a4f-4819-8943-5fe88488888c-kube-api-access-xtl8n\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.375848 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.371409    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.375848 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.372118    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b6a788-2a4f-4819-8943-5fe88488888c-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.375848 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.372364    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.375848 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.373915    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.382796 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.382769    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.411506 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.411105    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.432906 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.427076    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtl8n\" (UniqueName: \"kubernetes.io/projected/d5b6a788-2a4f-4819-8943-5fe88488888c-kube-api-access-xtl8n\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.456121 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.455780    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.512769 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.479084    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:35.623726 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.623681    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:17:35.712624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:35.711869    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:36.452423 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:36.452394    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"]
Nov 03 20:17:36.662837 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:36.647637    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:17:36.662837 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:36.647735    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:17:36.662837 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:36.647820    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:17:37.470671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:37.470614    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh" event={"ID":"d5b6a788-2a4f-4819-8943-5fe88488888c","Type":"ContainerStarted","Data":"2410dac0bb78878e260f747d14b506782b5fcd409a9a7e426c670a22552926d0"}
Nov 03 20:17:37.471334 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:37.471308    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh" event={"ID":"d5b6a788-2a4f-4819-8943-5fe88488888c","Type":"ContainerStarted","Data":"d3a316fddcb39b4641241c14ede46cb19a589b144f49a527c2cc9802d0227ec2"}
Nov 03 20:17:37.500122 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:37.500061    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh" podStartSLOduration=3.500041855 podStartE2EDuration="3.500041855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:37.496246501 +0000 UTC m=+987.629374970" watchObservedRunningTime="2026-11-03 20:17:37.500041855 +0000 UTC m=+987.633170318"
Nov 03 20:17:37.714067 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:37.714038    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:37.714398 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:37.714374    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 20:17:37.714673 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:37.714641    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh" podUID="d5b6a788-2a4f-4819-8943-5fe88488888c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 20:17:37.898560 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:37.898455    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw"]
Nov 03 20:17:37.931449 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:37.931410    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw"
Nov 03 20:17:37.973615 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:37.973556    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw"]
Nov 03 20:17:38.111634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:38.059650    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f34b3118-c425-4b46-87a8-b22d7a101c6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw\" (UID: \"f34b3118-c425-4b46-87a8-b22d7a101c6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw"
Nov 03 20:17:38.218755 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:38.217751    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz8fb\" (UniqueName: \"kubernetes.io/projected/f34b3118-c425-4b46-87a8-b22d7a101c6f-kube-api-access-bz8fb\") pod \"payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw\" (UID: \"f34b3118-c425-4b46-87a8-b22d7a101c6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw"
Nov 03 20:17:38.218755 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:38.158565    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-18ba46-dockercfg-m7qhr\""
Nov 03 20:17:38.218755 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:38.158411    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-18ba46\""
Nov 03 20:17:38.218755 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:38.195437    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9"]
Nov 03 20:17:38.268848 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:38.268814    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9"]
Nov 03 20:17:38.269172 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:38.269155    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9"
Nov 03 20:17:38.319453 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:38.319423    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz8fb\" (UniqueName: \"kubernetes.io/projected/f34b3118-c425-4b46-87a8-b22d7a101c6f-kube-api-access-bz8fb\") pod \"payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw\" (UID: \"f34b3118-c425-4b46-87a8-b22d7a101c6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw"
Nov 03 20:17:38.319674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:38.319539    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799-plugins-config-volume\") pod \"payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9\" (UID: \"cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799\") " pod="openshift-ingress/payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9"
Nov 03 20:17:38.319674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:38.319581    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j87m8\" (UniqueName: \"kubernetes.io/projected/cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799-kube-api-access-j87m8\") pod \"payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9\" (UID: \"cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799\") " pod="openshift-ingress/payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9"
Nov 03 20:17:38.319674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:38.319623    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f34b3118-c425-4b46-87a8-b22d7a101c6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw\" (UID: \"f34b3118-c425-4b46-87a8-b22d7a101c6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw"
Nov 03 20:17:38.320472 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:38.320446    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f34b3118-c425-4b46-87a8-b22d7a101c6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw\" (UID: \"f34b3118-c425-4b46-87a8-b22d7a101c6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw"
Nov 03 20:17:38.340646 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:38.340585    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz8fb\" (UniqueName: \"kubernetes.io/projected/f34b3118-c425-4b46-87a8-b22d7a101c6f-kube-api-access-bz8fb\") pod \"payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw\" (UID: \"f34b3118-c425-4b46-87a8-b22d7a101c6f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw"
Nov 03 20:17:38.423623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:38.421253    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799-plugins-config-volume\") pod \"payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9\" (UID: \"cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799\") " pod="openshift-ingress/payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9"
Nov 03 20:17:38.423623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:38.421332    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j87m8\" (UniqueName: \"kubernetes.io/projected/cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799-kube-api-access-j87m8\") pod \"payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9\" (UID: \"cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799\") " pod="openshift-ingress/payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9"
Nov 03 20:17:38.423623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:38.423235    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799-plugins-config-volume\") pod \"payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9\" (UID: \"cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799\") " pod="openshift-ingress/payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9"
Nov 03 20:17:38.463666 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:38.461285    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j87m8\" (UniqueName: \"kubernetes.io/projected/cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799-kube-api-access-j87m8\") pod \"payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9\" (UID: \"cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799\") " pod="openshift-ingress/payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9"
Nov 03 20:17:38.555676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:38.544949    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw"
Nov 03 20:17:38.595386 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:38.595272    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9"
Nov 03 20:17:38.712960 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:38.712929    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 20:17:38.713056 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:38.712997    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh" podUID="d5b6a788-2a4f-4819-8943-5fe88488888c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 20:17:38.870705 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:38.870670    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw"]
Nov 03 20:17:38.871464 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:17:38.871430    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34b3118_c425_4b46_87a8_b22d7a101c6f.slice/crio-7b2b098d025938a4a513245ca203a38d90980f9b1fa59b862b7d76560d7ab6eb WatchSource:0}: Error finding container 7b2b098d025938a4a513245ca203a38d90980f9b1fa59b862b7d76560d7ab6eb: Status 404 returned error can't find the container with id 7b2b098d025938a4a513245ca203a38d90980f9b1fa59b862b7d76560d7ab6eb
Nov 03 20:17:38.933214 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:38.932290    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9"]
Nov 03 20:17:38.934956 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:17:38.934924    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf9fbd7f_d1ef_4d77_8f1d_d8c6e64cd799.slice/crio-5e79932eda8cdd9d693aa8d33cd97b1bbc5ec832390cd00555f3020ab9b0ed44 WatchSource:0}: Error finding container 5e79932eda8cdd9d693aa8d33cd97b1bbc5ec832390cd00555f3020ab9b0ed44: Status 404 returned error can't find the container with id 5e79932eda8cdd9d693aa8d33cd97b1bbc5ec832390cd00555f3020ab9b0ed44
Nov 03 20:17:39.502334 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:39.502294    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw" event={"ID":"f34b3118-c425-4b46-87a8-b22d7a101c6f","Type":"ContainerStarted","Data":"dbe9c3553f5c048870cf1dc5e797e84eca41314351c267f7aecb904d891afcfb"}
Nov 03 20:17:39.502523 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:39.502363    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw" event={"ID":"f34b3118-c425-4b46-87a8-b22d7a101c6f","Type":"ContainerStarted","Data":"7b2b098d025938a4a513245ca203a38d90980f9b1fa59b862b7d76560d7ab6eb"}
Nov 03 20:17:39.502523 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:39.502401    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw"
Nov 03 20:17:39.504319 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:39.504295    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9" event={"ID":"cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799","Type":"ContainerStarted","Data":"5dd0ee0556afec28673665ff12fbc9a052d911726b87e6abfd158dc5aed0a8da"}
Nov 03 20:17:39.504450 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:39.504341    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9" event={"ID":"cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799","Type":"ContainerStarted","Data":"5e79932eda8cdd9d693aa8d33cd97b1bbc5ec832390cd00555f3020ab9b0ed44"}
Nov 03 20:17:39.504450 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:39.504370    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9"
Nov 03 20:17:39.525106 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:39.525049    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw" podStartSLOduration=2.525030695 podStartE2EDuration="2.525030695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:39.52030722 +0000 UTC m=+989.653435683" watchObservedRunningTime="2026-11-03 20:17:39.525030695 +0000 UTC m=+989.658159158"
Nov 03 20:17:39.715050 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:39.715013    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 20:17:39.715896 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:39.715861    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh" podUID="d5b6a788-2a4f-4819-8943-5fe88488888c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 20:17:40.713391 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:40.713354    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 20:17:40.713573 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:40.713411    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh" podUID="d5b6a788-2a4f-4819-8943-5fe88488888c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 20:17:41.713370 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:41.713331    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 20:17:41.713787 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:41.713396    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh" podUID="d5b6a788-2a4f-4819-8943-5fe88488888c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 20:17:42.713045 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:42.713007    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 20:17:42.713325 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:42.713297    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh" podUID="d5b6a788-2a4f-4819-8943-5fe88488888c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 20:17:42.875631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:42.874464    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9" podStartSLOduration=5.874444662 podStartE2EDuration="5.874444662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:39.538915097 +0000 UTC m=+989.672043560" watchObservedRunningTime="2026-11-03 20:17:42.874444662 +0000 UTC m=+993.007573125"
Nov 03 20:17:42.882055 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:42.881979    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd"]
Nov 03 20:17:42.890915 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:42.887038    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd"]
Nov 03 20:17:42.890915 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:42.887199    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd"
Nov 03 20:17:42.897468 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:42.896336    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-18ba46-serving-cert\""
Nov 03 20:17:43.099690 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:43.098943    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgghx\" (UniqueName: \"kubernetes.io/projected/b5080708-ea89-4bbd-a33c-d74e9d301230-kube-api-access-cgghx\") pod \"maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd\" (UID: \"b5080708-ea89-4bbd-a33c-d74e9d301230\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd"
Nov 03 20:17:43.108989 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:43.108959    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5080708-ea89-4bbd-a33c-d74e9d301230-maas-api-tls\") pod \"maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd\" (UID: \"b5080708-ea89-4bbd-a33c-d74e9d301230\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd"
Nov 03 20:17:43.109414 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:43.109385    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5080708-ea89-4bbd-a33c-d74e9d301230-metrics-certs\") pod \"maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd\" (UID: \"b5080708-ea89-4bbd-a33c-d74e9d301230\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd"
Nov 03 20:17:43.109654 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:43.109632    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5080708-ea89-4bbd-a33c-d74e9d301230-openshift-service-ca\") pod \"maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd\" (UID: \"b5080708-ea89-4bbd-a33c-d74e9d301230\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd"
Nov 03 20:17:43.211515 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:43.211482    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5080708-ea89-4bbd-a33c-d74e9d301230-metrics-certs\") pod \"maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd\" (UID: \"b5080708-ea89-4bbd-a33c-d74e9d301230\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd"
Nov 03 20:17:43.211515 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:43.211529    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5080708-ea89-4bbd-a33c-d74e9d301230-openshift-service-ca\") pod \"maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd\" (UID: \"b5080708-ea89-4bbd-a33c-d74e9d301230\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd"
Nov 03 20:17:43.211763 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:43.211623    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgghx\" (UniqueName: \"kubernetes.io/projected/b5080708-ea89-4bbd-a33c-d74e9d301230-kube-api-access-cgghx\") pod \"maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd\" (UID: \"b5080708-ea89-4bbd-a33c-d74e9d301230\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd"
Nov 03 20:17:43.212163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:43.211797    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5080708-ea89-4bbd-a33c-d74e9d301230-maas-api-tls\") pod \"maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd\" (UID: \"b5080708-ea89-4bbd-a33c-d74e9d301230\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd"
Nov 03 20:17:43.212394 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:43.212370    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5080708-ea89-4bbd-a33c-d74e9d301230-openshift-service-ca\") pod \"maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd\" (UID: \"b5080708-ea89-4bbd-a33c-d74e9d301230\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd"
Nov 03 20:17:43.214283 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:43.214255    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5080708-ea89-4bbd-a33c-d74e9d301230-maas-api-tls\") pod \"maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd\" (UID: \"b5080708-ea89-4bbd-a33c-d74e9d301230\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd"
Nov 03 20:17:43.214432 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:43.214410    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5080708-ea89-4bbd-a33c-d74e9d301230-metrics-certs\") pod \"maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd\" (UID: \"b5080708-ea89-4bbd-a33c-d74e9d301230\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd"
Nov 03 20:17:43.223417 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:43.223397    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgghx\" (UniqueName: \"kubernetes.io/projected/b5080708-ea89-4bbd-a33c-d74e9d301230-kube-api-access-cgghx\") pod \"maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd\" (UID: \"b5080708-ea89-4bbd-a33c-d74e9d301230\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd"
Nov 03 20:17:43.551938 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:43.545984    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd"
Nov 03 20:17:43.713785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:43.713149    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 20:17:43.713785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:43.713232    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh" podUID="d5b6a788-2a4f-4819-8943-5fe88488888c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 20:17:43.871625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:43.871574    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd"]
Nov 03 20:17:43.873197 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:17:43.873160    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5080708_ea89_4bbd_a33c_d74e9d301230.slice/crio-d998a42cdd6f601152a33edee92f60318661473d4a4bc82e83ca3a4da301aad9 WatchSource:0}: Error finding container d998a42cdd6f601152a33edee92f60318661473d4a4bc82e83ca3a4da301aad9: Status 404 returned error can't find the container with id d998a42cdd6f601152a33edee92f60318661473d4a4bc82e83ca3a4da301aad9
Nov 03 20:17:43.875375 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:43.875357    4421 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:17:44.588716 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:44.588680    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd" event={"ID":"b5080708-ea89-4bbd-a33c-d74e9d301230","Type":"ContainerStarted","Data":"726a97c4135c46e615f734ed0fa8f68d228481b5595b1cec3b04cf345f0d26c8"}
Nov 03 20:17:44.589125 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:44.588730    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd" event={"ID":"b5080708-ea89-4bbd-a33c-d74e9d301230","Type":"ContainerStarted","Data":"d998a42cdd6f601152a33edee92f60318661473d4a4bc82e83ca3a4da301aad9"}
Nov 03 20:17:44.589125 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:44.588835    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd"
Nov 03 20:17:44.610982 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:44.610925    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd" podStartSLOduration=2.090884756 podStartE2EDuration="2.610908952s" totalImagesPullingTime="520.024196ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:44.60985064 +0000 UTC m=+994.742979103" watchObservedRunningTime="2026-11-03 20:17:44.610908952 +0000 UTC m=+994.744037419"
Nov 03 20:17:45.231737 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:45.231704    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:45.232162 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:45.232134    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:45.233056 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:45.233036    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:17:49.629339 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:17:49.629293    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:17:49.720309 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.720280    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"]
Nov 03 20:17:49.725081 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.725060    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:49.744049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.741920    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"]
Nov 03 20:17:49.860191 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.860154    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:49.860369 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.860220    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:49.860369 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.860287    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psv4p\" (UniqueName: \"kubernetes.io/projected/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-kube-api-access-psv4p\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:49.860369 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.860325    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:49.860369 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.860342    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:49.860560 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.860471    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:49.860560 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.860489    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:49.860560 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.860521    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:49.860730 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.860713    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:49.962026 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.961995    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:49.962212 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.962099    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:49.962212 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.962169    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:49.962212 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.962191    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psv4p\" (UniqueName: \"kubernetes.io/projected/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-kube-api-access-psv4p\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:49.962388 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.962224    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:49.962388 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.962249    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:49.962388 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.962318    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:49.962388 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.962344    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:49.962585 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.962381    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:49.962585 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.962396    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:49.962730 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.962705    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:49.962790 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.962726    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:49.962857 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.962838    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:49.963067 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.963046    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:49.968620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.964927    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:49.968620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.965331    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:49.977458 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.977431    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:49.977765 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:49.977741    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psv4p\" (UniqueName: \"kubernetes.io/projected/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-kube-api-access-psv4p\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:50.045771 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:50.045734    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:50.346272 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:50.346172    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"]
Nov 03 20:17:50.408388 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:50.408335    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:17:50.408525 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:50.408433    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:17:50.408525 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:50.408475    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:17:50.515282 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:50.515258    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9"
Nov 03 20:17:50.515438 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:50.515340    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw"
Nov 03 20:17:50.606724 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:50.606633    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd"
Nov 03 20:17:50.664235 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:50.664207    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr" event={"ID":"bac03c56-aa2e-4ba9-9841-4a9122b7fddb","Type":"ContainerStarted","Data":"6e897b58d48b0137a0ac8f1579c6ed61bfd9f2dff70e17e3780c204a4977ad6c"}
Nov 03 20:17:50.664706 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:50.664678    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr" event={"ID":"bac03c56-aa2e-4ba9-9841-4a9122b7fddb","Type":"ContainerStarted","Data":"05d1014172d9401bee50e68878a3f97933b9effbadf4c648265217a99d2a7363"}
Nov 03 20:17:50.689628 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:50.689532    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr" podStartSLOduration=1.689513347 podStartE2EDuration="1.689513347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:50.687494133 +0000 UTC m=+1000.820622612" watchObservedRunningTime="2026-11-03 20:17:50.689513347 +0000 UTC m=+1000.822641811"
Nov 03 20:17:51.046629 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:51.046590    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:51.048355 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:51.048325    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 20:17:51.048481 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:51.048377    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr" podUID="bac03c56-aa2e-4ba9-9841-4a9122b7fddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 20:17:52.063624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:52.058704    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 20:17:52.063624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:52.058767    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr" podUID="bac03c56-aa2e-4ba9-9841-4a9122b7fddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 20:17:53.057004 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:53.056967    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 20:17:53.057299 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:53.057270    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr" podUID="bac03c56-aa2e-4ba9-9841-4a9122b7fddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 20:17:54.047345 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:54.047308    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 20:17:54.048331 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:54.048298    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr" podUID="bac03c56-aa2e-4ba9-9841-4a9122b7fddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 20:17:55.075659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:55.074033    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 20:17:55.075659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:55.074093    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr" podUID="bac03c56-aa2e-4ba9-9841-4a9122b7fddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 20:17:56.051651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:56.046943    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 20:17:56.051651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:56.047012    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr" podUID="bac03c56-aa2e-4ba9-9841-4a9122b7fddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 20:17:57.046276 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:57.046241    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 20:17:57.046683 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:57.046305    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr" podUID="bac03c56-aa2e-4ba9-9841-4a9122b7fddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 20:17:58.715918 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:58.715883    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:58.718429 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:58.718048    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:58.719114 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:58.719078    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:17:59.444425 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:59.444376    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp"]
Nov 03 20:17:59.448897 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:59.448875    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp"
Nov 03 20:17:59.462081 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:59.462054    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9b65143e\""
Nov 03 20:17:59.463187 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:59.463164    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp"]
Nov 03 20:17:59.594539 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:59.594501    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/734c6a7d-7e9d-4a9c-8577-263368974782-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp\" (UID: \"734c6a7d-7e9d-4a9c-8577-263368974782\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp"
Nov 03 20:17:59.594797 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:59.594550    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqx85\" (UniqueName: \"kubernetes.io/projected/734c6a7d-7e9d-4a9c-8577-263368974782-kube-api-access-sqx85\") pod \"payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp\" (UID: \"734c6a7d-7e9d-4a9c-8577-263368974782\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp"
Nov 03 20:17:59.708673 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:59.696925    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/734c6a7d-7e9d-4a9c-8577-263368974782-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp\" (UID: \"734c6a7d-7e9d-4a9c-8577-263368974782\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp"
Nov 03 20:17:59.708673 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:59.697000    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqx85\" (UniqueName: \"kubernetes.io/projected/734c6a7d-7e9d-4a9c-8577-263368974782-kube-api-access-sqx85\") pod \"payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp\" (UID: \"734c6a7d-7e9d-4a9c-8577-263368974782\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp"
Nov 03 20:17:59.708673 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:59.698385    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/734c6a7d-7e9d-4a9c-8577-263368974782-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp\" (UID: \"734c6a7d-7e9d-4a9c-8577-263368974782\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp"
Nov 03 20:17:59.749872 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:59.749838    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqx85\" (UniqueName: \"kubernetes.io/projected/734c6a7d-7e9d-4a9c-8577-263368974782-kube-api-access-sqx85\") pod \"payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp\" (UID: \"734c6a7d-7e9d-4a9c-8577-263368974782\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp"
Nov 03 20:17:59.772477 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:17:59.771270    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp"
Nov 03 20:18:00.140523 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:18:00.140493    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734c6a7d_7e9d_4a9c_8577_263368974782.slice/crio-5ef59e34f8adb172bc78ca95b25fb502cfdf77472242c530c4e7957c915d938d WatchSource:0}: Error finding container 5ef59e34f8adb172bc78ca95b25fb502cfdf77472242c530c4e7957c915d938d: Status 404 returned error can't find the container with id 5ef59e34f8adb172bc78ca95b25fb502cfdf77472242c530c4e7957c915d938d
Nov 03 20:18:00.141068 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:00.141044    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp"]
Nov 03 20:18:00.842291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:00.842254    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp" event={"ID":"734c6a7d-7e9d-4a9c-8577-263368974782","Type":"ContainerStarted","Data":"7e0eeece2348b8b34dca43e6bcf59e48d1fb95d655450761811a133a970b6423"}
Nov 03 20:18:00.842698 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:00.842330    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp" event={"ID":"734c6a7d-7e9d-4a9c-8577-263368974782","Type":"ContainerStarted","Data":"5ef59e34f8adb172bc78ca95b25fb502cfdf77472242c530c4e7957c915d938d"}
Nov 03 20:18:00.842698 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:00.842505    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp"
Nov 03 20:18:00.862180 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:00.862131    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp" podStartSLOduration=1.862117794 podStartE2EDuration="1.862117794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:00.860074675 +0000 UTC m=+1010.993203138" watchObservedRunningTime="2026-11-03 20:18:00.862117794 +0000 UTC m=+1010.995246290"
Nov 03 20:18:01.801321 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:01.801285    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"]
Nov 03 20:18:01.805855 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:01.805832    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:01.830641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:01.830403    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"]
Nov 03 20:18:01.900585 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:01.900556    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-18ba46-openshift-default-dockercfg-zvvnj\""
Nov 03 20:18:01.943590 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:01.943553    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-token\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:01.943759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:01.943617    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-workload-certs\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:01.943802 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:01.943765    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bx5j\" (UniqueName: \"kubernetes.io/projected/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-kube-api-access-5bx5j\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:01.943838 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:01.943809    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-workload-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:01.943874 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:01.943836    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-envoy\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:01.944031 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:01.944015    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istiod-ca-cert\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:01.944076 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:01.944060    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-credential-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:01.944216 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:01.944195    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-data\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:01.944372 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:01.944357    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-podinfo\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:02.045716 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:02.045682    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istiod-ca-cert\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:02.045927 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:02.045733    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-credential-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:02.045927 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:02.045806    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-data\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:02.045927 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:02.045912    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-podinfo\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:02.046096 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:02.045998    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-token\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:02.046096 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:02.046016    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-workload-certs\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:02.046096 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:02.046072    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bx5j\" (UniqueName: \"kubernetes.io/projected/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-kube-api-access-5bx5j\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:02.046096 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:02.046093    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-workload-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:02.046292 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:02.046112    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-envoy\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:02.046351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:02.046299    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-data\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:02.046422 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:02.046397    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-workload-certs\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:02.046536 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:02.046507    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-credential-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:02.046536 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:02.046526    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istiod-ca-cert\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:02.046785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:02.046728    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-workload-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:02.048799 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:02.048774    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-podinfo\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:02.048903 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:02.048889    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-envoy\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:02.058881 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:02.058809    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bx5j\" (UniqueName: \"kubernetes.io/projected/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-kube-api-access-5bx5j\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:02.059007 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:02.058874    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-token\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:02.140443 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:02.140406    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:02.464313 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:02.464286    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"]
Nov 03 20:18:02.465037 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:18:02.465011    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3ad6df8_70a3_4dd1_a5e7_fc79003fb609.slice/crio-2500cb78568060a8615177bdc66ff2feddf5e8b9c1fda985507bba3aec6d830d WatchSource:0}: Error finding container 2500cb78568060a8615177bdc66ff2feddf5e8b9c1fda985507bba3aec6d830d: Status 404 returned error can't find the container with id 2500cb78568060a8615177bdc66ff2feddf5e8b9c1fda985507bba3aec6d830d
Nov 03 20:18:02.467939 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:02.467900    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:02.468028 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:02.467969    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:02.468028 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:02.468006    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:02.892959 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:02.884206    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4" event={"ID":"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609","Type":"ContainerStarted","Data":"36d8082d42a2869522600c3c25b1b7d66ed747096e7d2171c6725857561d8189"}
Nov 03 20:18:02.892959 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:02.884278    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4" event={"ID":"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609","Type":"ContainerStarted","Data":"2500cb78568060a8615177bdc66ff2feddf5e8b9c1fda985507bba3aec6d830d"}
Nov 03 20:18:02.924621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:02.922562    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4" podStartSLOduration=1.9225403540000001 podStartE2EDuration="1.922540354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:02.915173851 +0000 UTC m=+1013.048302340" watchObservedRunningTime="2026-11-03 20:18:02.922540354 +0000 UTC m=+1013.055668821"
Nov 03 20:18:03.151020 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:03.150930    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:03.165703 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:03.161727    4421 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 03 20:18:03.165703 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:03.161802    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4" podUID="f3ad6df8-70a3-4dd1-a5e7-fc79003fb609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 03 20:18:04.154623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:04.149980    4421 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 03 20:18:04.154623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:04.150051    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4" podUID="f3ad6df8-70a3-4dd1-a5e7-fc79003fb609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 03 20:18:05.151624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:05.146991    4421 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 03 20:18:05.151624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:05.147048    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4" podUID="f3ad6df8-70a3-4dd1-a5e7-fc79003fb609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 03 20:18:06.155625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:06.145980    4421 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 03 20:18:06.155625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:06.146049    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4" podUID="f3ad6df8-70a3-4dd1-a5e7-fc79003fb609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 03 20:18:07.151628 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:07.151370    4421 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 03 20:18:07.151628 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:07.151431    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4" podUID="f3ad6df8-70a3-4dd1-a5e7-fc79003fb609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 03 20:18:08.159627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:08.144973    4421 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 03 20:18:08.159627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:08.145044    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4" podUID="f3ad6df8-70a3-4dd1-a5e7-fc79003fb609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 03 20:18:09.146702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:09.146674    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:09.147171 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:09.147155    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:09.147817 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:09.147800    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:09.597934 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:09.597899    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-db5557-ffc78867d-sggqh"]
Nov 03 20:18:11.642622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:11.642565    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l"]
Nov 03 20:18:11.678196 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:11.678168    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l"]
Nov 03 20:18:11.678437 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:11.678262    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l"
Nov 03 20:18:11.687250 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:11.687217    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-98148a\""
Nov 03 20:18:11.839650 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:11.839591    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs2ls\" (UniqueName: \"kubernetes.io/projected/f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be-kube-api-access-fs2ls\") pod \"payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l\" (UID: \"f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be\") " pod="openshift-ingress/payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l"
Nov 03 20:18:11.840172 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:11.840148    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be-plugins-config-volume\") pod \"payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l\" (UID: \"f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be\") " pod="openshift-ingress/payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l"
Nov 03 20:18:11.888854 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:11.888826    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp"
Nov 03 20:18:11.952643 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:11.942923    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be-plugins-config-volume\") pod \"payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l\" (UID: \"f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be\") " pod="openshift-ingress/payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l"
Nov 03 20:18:11.976886 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:11.976853    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs2ls\" (UniqueName: \"kubernetes.io/projected/f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be-kube-api-access-fs2ls\") pod \"payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l\" (UID: \"f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be\") " pod="openshift-ingress/payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l"
Nov 03 20:18:12.018621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:11.953946    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be-plugins-config-volume\") pod \"payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l\" (UID: \"f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be\") " pod="openshift-ingress/payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l"
Nov 03 20:18:12.074694 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:12.074591    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs2ls\" (UniqueName: \"kubernetes.io/projected/f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be-kube-api-access-fs2ls\") pod \"payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l\" (UID: \"f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be\") " pod="openshift-ingress/payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l"
Nov 03 20:18:12.299301 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:12.299160    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l"
Nov 03 20:18:12.603903 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:12.603758    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l"]
Nov 03 20:18:12.610494 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:18:12.610466    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf583fa6c_ff3a_4af6_8b3f_2e3de29fa4be.slice/crio-42743822f858a62374b18a5cee442e28c53c56301d25c370c45b5808d0c83fb7 WatchSource:0}: Error finding container 42743822f858a62374b18a5cee442e28c53c56301d25c370c45b5808d0c83fb7: Status 404 returned error can't find the container with id 42743822f858a62374b18a5cee442e28c53c56301d25c370c45b5808d0c83fb7
Nov 03 20:18:13.157152 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:13.157119    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l" event={"ID":"f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be","Type":"ContainerStarted","Data":"21ac8bc4582349cd23f23d4956eda0a0a1279dc23cc879b0f744ae04cb20999a"}
Nov 03 20:18:13.157528 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:13.157173    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l" event={"ID":"f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be","Type":"ContainerStarted","Data":"42743822f858a62374b18a5cee442e28c53c56301d25c370c45b5808d0c83fb7"}
Nov 03 20:18:13.157528 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:13.157242    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l"
Nov 03 20:18:13.177437 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:13.177391    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l" podStartSLOduration=2.177377298 podStartE2EDuration="2.177377298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:13.175023303 +0000 UTC m=+1023.308151765" watchObservedRunningTime="2026-11-03 20:18:13.177377298 +0000 UTC m=+1023.310505761"
Nov 03 20:18:14.598237 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:14.598168    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-db5557-ffc78867d-sggqh" podUID="c032b61d-02e3-4c2d-b7e7-2768ec0c62cf" containerName="payload-processing" containerID="cri-o://f946f087823ccc5e293edc86f145d53bc6f6af507fbd99e1a909af709cb44f94" gracePeriod=25
Nov 03 20:18:14.947809 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:14.947779    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-db5557-ffc78867d-sggqh"
Nov 03 20:18:15.018302 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:15.018270    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c032b61d-02e3-4c2d-b7e7-2768ec0c62cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c032b61d-02e3-4c2d-b7e7-2768ec0c62cf-plugins-config-volume\") pod \"c032b61d-02e3-4c2d-b7e7-2768ec0c62cf\" (UID: \"c032b61d-02e3-4c2d-b7e7-2768ec0c62cf\") "
Nov 03 20:18:15.018492 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:15.018365    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c032b61d-02e3-4c2d-b7e7-2768ec0c62cf-kube-api-access-wg2qw\" (UniqueName: \"kubernetes.io/projected/c032b61d-02e3-4c2d-b7e7-2768ec0c62cf-kube-api-access-wg2qw\") pod \"c032b61d-02e3-4c2d-b7e7-2768ec0c62cf\" (UID: \"c032b61d-02e3-4c2d-b7e7-2768ec0c62cf\") "
Nov 03 20:18:15.018770 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:15.018742    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c032b61d-02e3-4c2d-b7e7-2768ec0c62cf-plugins-config-volume" pod "c032b61d-02e3-4c2d-b7e7-2768ec0c62cf" (UID: "c032b61d-02e3-4c2d-b7e7-2768ec0c62cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:15.018973 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:15.018954    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c032b61d-02e3-4c2d-b7e7-2768ec0c62cf-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:15.021337 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:15.021308    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c032b61d-02e3-4c2d-b7e7-2768ec0c62cf-kube-api-access-wg2qw" pod "c032b61d-02e3-4c2d-b7e7-2768ec0c62cf" (UID: "c032b61d-02e3-4c2d-b7e7-2768ec0c62cf"). InnerVolumeSpecName "kube-api-access-wg2qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:15.120536 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:15.120490    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg2qw\" (UniqueName: \"kubernetes.io/projected/c032b61d-02e3-4c2d-b7e7-2768ec0c62cf-kube-api-access-wg2qw\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:15.171028 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:15.170957    4421 generic.go:356] "Generic (PLEG): container finished" podID="c032b61d-02e3-4c2d-b7e7-2768ec0c62cf" containerID="f946f087823ccc5e293edc86f145d53bc6f6af507fbd99e1a909af709cb44f94" exitCode=0
Nov 03 20:18:15.171028 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:15.171001    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-db5557-ffc78867d-sggqh"
Nov 03 20:18:15.171028 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:15.171007    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-db5557-ffc78867d-sggqh" event={"ID":"c032b61d-02e3-4c2d-b7e7-2768ec0c62cf","Type":"ContainerDied","Data":"f946f087823ccc5e293edc86f145d53bc6f6af507fbd99e1a909af709cb44f94"}
Nov 03 20:18:15.171264 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:15.171049    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-db5557-ffc78867d-sggqh" event={"ID":"c032b61d-02e3-4c2d-b7e7-2768ec0c62cf","Type":"ContainerDied","Data":"95d5c0cb4fdca2a0304fddab134032b3f6f31bf8d81f2639b4b525b7dd8958a1"}
Nov 03 20:18:15.171264 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:15.171069    4421 scope.go:122] "RemoveContainer" containerID="f946f087823ccc5e293edc86f145d53bc6f6af507fbd99e1a909af709cb44f94"
Nov 03 20:18:15.181805 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:15.181787    4421 scope.go:122] "RemoveContainer" containerID="f946f087823ccc5e293edc86f145d53bc6f6af507fbd99e1a909af709cb44f94"
Nov 03 20:18:15.182097 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:18:15.182077    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f946f087823ccc5e293edc86f145d53bc6f6af507fbd99e1a909af709cb44f94\": container with ID starting with f946f087823ccc5e293edc86f145d53bc6f6af507fbd99e1a909af709cb44f94 not found: ID does not exist" containerID="f946f087823ccc5e293edc86f145d53bc6f6af507fbd99e1a909af709cb44f94"
Nov 03 20:18:15.182199 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:15.182102    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f946f087823ccc5e293edc86f145d53bc6f6af507fbd99e1a909af709cb44f94"} err="failed to get container status \"f946f087823ccc5e293edc86f145d53bc6f6af507fbd99e1a909af709cb44f94\": rpc error: code = NotFound desc = could not find container \"f946f087823ccc5e293edc86f145d53bc6f6af507fbd99e1a909af709cb44f94\": container with ID starting with f946f087823ccc5e293edc86f145d53bc6f6af507fbd99e1a909af709cb44f94 not found: ID does not exist"
Nov 03 20:18:15.201170 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:15.201149    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-db5557-ffc78867d-sggqh"]
Nov 03 20:18:15.207082 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:15.207059    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-db5557-ffc78867d-sggqh"]
Nov 03 20:18:16.017008 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:16.016980    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c032b61d-02e3-4c2d-b7e7-2768ec0c62cf" path="/var/lib/kubelet/pods/c032b61d-02e3-4c2d-b7e7-2768ec0c62cf/volumes"
Nov 03 20:18:17.330967 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:17.330291    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d"]
Nov 03 20:18:17.332547 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:17.332510    4421 cpu_manager.go:414] "containerMap: removing container" podUID="c032b61d-02e3-4c2d-b7e7-2768ec0c62cf" containerName="payload-processing"
Nov 03 20:18:17.332547 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:17.332542    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c032b61d-02e3-4c2d-b7e7-2768ec0c62cf" containerName="payload-processing"
Nov 03 20:18:17.332779 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:17.332749    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c032b61d-02e3-4c2d-b7e7-2768ec0c62cf" containerName="payload-processing"
Nov 03 20:18:17.347617 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:17.338515    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d"
Nov 03 20:18:17.347617 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:17.344772    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-98148a-serving-cert\""
Nov 03 20:18:17.351741 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:17.348661    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d"]
Nov 03 20:18:17.554627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:17.551194    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-maas-api-tls\") pod \"maas-api-e2e-api-a-98148a-6576c6f578-9kc9d\" (UID: \"264eaf78-ec83-4872-8a0b-e1de8f8e5a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d"
Nov 03 20:18:17.554627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:17.551534    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt6nb\" (UniqueName: \"kubernetes.io/projected/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-kube-api-access-bt6nb\") pod \"maas-api-e2e-api-a-98148a-6576c6f578-9kc9d\" (UID: \"264eaf78-ec83-4872-8a0b-e1de8f8e5a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d"
Nov 03 20:18:17.554627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:17.551834    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-metrics-certs\") pod \"maas-api-e2e-api-a-98148a-6576c6f578-9kc9d\" (UID: \"264eaf78-ec83-4872-8a0b-e1de8f8e5a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d"
Nov 03 20:18:17.554627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:17.552038    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-openshift-service-ca\") pod \"maas-api-e2e-api-a-98148a-6576c6f578-9kc9d\" (UID: \"264eaf78-ec83-4872-8a0b-e1de8f8e5a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d"
Nov 03 20:18:17.708949 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:17.708911    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt6nb\" (UniqueName: \"kubernetes.io/projected/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-kube-api-access-bt6nb\") pod \"maas-api-e2e-api-a-98148a-6576c6f578-9kc9d\" (UID: \"264eaf78-ec83-4872-8a0b-e1de8f8e5a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d"
Nov 03 20:18:17.709121 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:17.708986    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-metrics-certs\") pod \"maas-api-e2e-api-a-98148a-6576c6f578-9kc9d\" (UID: \"264eaf78-ec83-4872-8a0b-e1de8f8e5a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d"
Nov 03 20:18:17.709121 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:17.709109    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-openshift-service-ca\") pod \"maas-api-e2e-api-a-98148a-6576c6f578-9kc9d\" (UID: \"264eaf78-ec83-4872-8a0b-e1de8f8e5a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d"
Nov 03 20:18:17.709203 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:17.709157    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-maas-api-tls\") pod \"maas-api-e2e-api-a-98148a-6576c6f578-9kc9d\" (UID: \"264eaf78-ec83-4872-8a0b-e1de8f8e5a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d"
Nov 03 20:18:17.709903 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:17.709815    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-openshift-service-ca\") pod \"maas-api-e2e-api-a-98148a-6576c6f578-9kc9d\" (UID: \"264eaf78-ec83-4872-8a0b-e1de8f8e5a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d"
Nov 03 20:18:17.711650 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:17.711631    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-maas-api-tls\") pod \"maas-api-e2e-api-a-98148a-6576c6f578-9kc9d\" (UID: \"264eaf78-ec83-4872-8a0b-e1de8f8e5a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d"
Nov 03 20:18:17.711737 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:17.711632    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-metrics-certs\") pod \"maas-api-e2e-api-a-98148a-6576c6f578-9kc9d\" (UID: \"264eaf78-ec83-4872-8a0b-e1de8f8e5a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d"
Nov 03 20:18:17.721777 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:17.721748    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt6nb\" (UniqueName: \"kubernetes.io/projected/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-kube-api-access-bt6nb\") pod \"maas-api-e2e-api-a-98148a-6576c6f578-9kc9d\" (UID: \"264eaf78-ec83-4872-8a0b-e1de8f8e5a0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d"
Nov 03 20:18:17.970676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:17.969697    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d"
Nov 03 20:18:18.270160 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:18.270123    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d"]
Nov 03 20:18:18.271297 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:18:18.271265    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264eaf78_ec83_4872_8a0b_e1de8f8e5a0b.slice/crio-7cbd2a86c780ab6df20985994fb3132c9fd0f4472d758f8a56caa9548fdf96d9 WatchSource:0}: Error finding container 7cbd2a86c780ab6df20985994fb3132c9fd0f4472d758f8a56caa9548fdf96d9: Status 404 returned error can't find the container with id 7cbd2a86c780ab6df20985994fb3132c9fd0f4472d758f8a56caa9548fdf96d9
Nov 03 20:18:19.342624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:19.331912    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d" event={"ID":"264eaf78-ec83-4872-8a0b-e1de8f8e5a0b","Type":"ContainerStarted","Data":"9c2b57a363542bc4542e31143ff641adbc3ee536f68b7fef147ed3667c7ece35"}
Nov 03 20:18:19.342624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:19.331982    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d" event={"ID":"264eaf78-ec83-4872-8a0b-e1de8f8e5a0b","Type":"ContainerStarted","Data":"7cbd2a86c780ab6df20985994fb3132c9fd0f4472d758f8a56caa9548fdf96d9"}
Nov 03 20:18:19.342624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:19.333105    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d"
Nov 03 20:18:19.365553 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:19.365490    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d" podStartSLOduration=1.7641817130000002 podStartE2EDuration="2.365474443s" totalImagesPullingTime="601.29273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:19.365079496 +0000 UTC m=+1029.498207972" watchObservedRunningTime="2026-11-03 20:18:19.365474443 +0000 UTC m=+1029.498602907"
Nov 03 20:18:19.731622 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:18:19.731562    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:18:21.706031 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:21.705997    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:18:21.706578 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:21.706550    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47" podUID="b270c5c6-0444-4410-a70d-37edb799adce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:22.616296 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:22.616253    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:23.802230 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:23.800643    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9"]
Nov 03 20:18:23.930917 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:23.927803    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw"]
Nov 03 20:18:24.202923 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:24.202892    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l"
Nov 03 20:18:26.900968 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:26.900932    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d"
Nov 03 20:18:28.801111 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:28.801074    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9" podUID="cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799" containerName="payload-processing" containerID="cri-o://5dd0ee0556afec28673665ff12fbc9a052d911726b87e6abfd158dc5aed0a8da" gracePeriod=25
Nov 03 20:18:28.928672 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:28.928637    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw" podUID="f34b3118-c425-4b46-87a8-b22d7a101c6f" containerName="payload-pre-processing" containerID="cri-o://dbe9c3553f5c048870cf1dc5e797e84eca41314351c267f7aecb904d891afcfb" gracePeriod=25
Nov 03 20:18:29.127336 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:29.127306    4421 generic.go:356] "Generic (PLEG): container finished" podID="f34b3118-c425-4b46-87a8-b22d7a101c6f" containerID="dbe9c3553f5c048870cf1dc5e797e84eca41314351c267f7aecb904d891afcfb" exitCode=0
Nov 03 20:18:29.127678 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:29.127478    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw" event={"ID":"f34b3118-c425-4b46-87a8-b22d7a101c6f","Type":"ContainerDied","Data":"dbe9c3553f5c048870cf1dc5e797e84eca41314351c267f7aecb904d891afcfb"}
Nov 03 20:18:29.128935 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:29.128915    4421 generic.go:356] "Generic (PLEG): container finished" podID="cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799" containerID="5dd0ee0556afec28673665ff12fbc9a052d911726b87e6abfd158dc5aed0a8da" exitCode=0
Nov 03 20:18:29.129011 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:29.129000    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9" event={"ID":"cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799","Type":"ContainerDied","Data":"5dd0ee0556afec28673665ff12fbc9a052d911726b87e6abfd158dc5aed0a8da"}
Nov 03 20:18:29.129055 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:29.129031    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9" event={"ID":"cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799","Type":"ContainerDied","Data":"5e79932eda8cdd9d693aa8d33cd97b1bbc5ec832390cd00555f3020ab9b0ed44"}
Nov 03 20:18:29.129055 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:29.129047    4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e79932eda8cdd9d693aa8d33cd97b1bbc5ec832390cd00555f3020ab9b0ed44"
Nov 03 20:18:29.148192 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:29.148167    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9"
Nov 03 20:18:29.313661 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:29.312385    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799-plugins-config-volume\") pod \"cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799\" (UID: \"cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799\") "
Nov 03 20:18:29.313661 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:29.312706    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799-kube-api-access-j87m8\" (UniqueName: \"kubernetes.io/projected/cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799-kube-api-access-j87m8\") pod \"cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799\" (UID: \"cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799\") "
Nov 03 20:18:29.314781 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:29.314722    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799-plugins-config-volume" pod "cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799" (UID: "cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:29.333576 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:29.332259    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799-kube-api-access-j87m8" pod "cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799" (UID: "cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799"). InnerVolumeSpecName "kube-api-access-j87m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:29.358018 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:29.357997    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw"
Nov 03 20:18:29.414174 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:29.414146    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f34b3118-c425-4b46-87a8-b22d7a101c6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f34b3118-c425-4b46-87a8-b22d7a101c6f-plugins-config-volume\") pod \"f34b3118-c425-4b46-87a8-b22d7a101c6f\" (UID: \"f34b3118-c425-4b46-87a8-b22d7a101c6f\") "
Nov 03 20:18:29.414375 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:29.414212    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f34b3118-c425-4b46-87a8-b22d7a101c6f-kube-api-access-bz8fb\" (UniqueName: \"kubernetes.io/projected/f34b3118-c425-4b46-87a8-b22d7a101c6f-kube-api-access-bz8fb\") pod \"f34b3118-c425-4b46-87a8-b22d7a101c6f\" (UID: \"f34b3118-c425-4b46-87a8-b22d7a101c6f\") "
Nov 03 20:18:29.414649 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:29.414631    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j87m8\" (UniqueName: \"kubernetes.io/projected/cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799-kube-api-access-j87m8\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:29.414748 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:29.414653    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:29.414748 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:29.414693    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f34b3118-c425-4b46-87a8-b22d7a101c6f-plugins-config-volume" pod "f34b3118-c425-4b46-87a8-b22d7a101c6f" (UID: "f34b3118-c425-4b46-87a8-b22d7a101c6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:29.416722 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:29.416692    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f34b3118-c425-4b46-87a8-b22d7a101c6f-kube-api-access-bz8fb" pod "f34b3118-c425-4b46-87a8-b22d7a101c6f" (UID: "f34b3118-c425-4b46-87a8-b22d7a101c6f"). InnerVolumeSpecName "kube-api-access-bz8fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:29.515696 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:29.515649    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz8fb\" (UniqueName: \"kubernetes.io/projected/f34b3118-c425-4b46-87a8-b22d7a101c6f-kube-api-access-bz8fb\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:29.515696 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:29.515685    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f34b3118-c425-4b46-87a8-b22d7a101c6f-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:30.137201 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:30.137116    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9"
Nov 03 20:18:30.137201 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:30.137125    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw" event={"ID":"f34b3118-c425-4b46-87a8-b22d7a101c6f","Type":"ContainerDied","Data":"7b2b098d025938a4a513245ca203a38d90980f9b1fa59b862b7d76560d7ab6eb"}
Nov 03 20:18:30.137201 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:30.137182    4421 scope.go:122] "RemoveContainer" containerID="dbe9c3553f5c048870cf1dc5e797e84eca41314351c267f7aecb904d891afcfb"
Nov 03 20:18:30.137790 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:30.137116    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw"
Nov 03 20:18:30.157347 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:30.157322    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw"]
Nov 03 20:18:30.162500 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:30.162474    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18ba46-5fb596f495-7c6dw"]
Nov 03 20:18:30.175829 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:30.175805    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9"]
Nov 03 20:18:30.181152 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:30.181126    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18ba46-7b79f64d5b-tvsg9"]
Nov 03 20:18:30.417052 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:30.416966    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd"]
Nov 03 20:18:30.417560 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:30.417504    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd" podUID="b5080708-ea89-4bbd-a33c-d74e9d301230" containerName="maas-api" containerID="cri-o://726a97c4135c46e615f734ed0fa8f68d228481b5595b1cec3b04cf345f0d26c8" gracePeriod=30
Nov 03 20:18:30.611624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:30.602104    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd" podUID="b5080708-ea89-4bbd-a33c-d74e9d301230" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.86:8443/health\": dial tcp 10.134.0.86:8443: connect: connection refused"
Nov 03 20:18:31.019003 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.018974    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd"
Nov 03 20:18:31.050914 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.050881    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5080708-ea89-4bbd-a33c-d74e9d301230-kube-api-access-cgghx\" (UniqueName: \"kubernetes.io/projected/b5080708-ea89-4bbd-a33c-d74e9d301230-kube-api-access-cgghx\") pod \"b5080708-ea89-4bbd-a33c-d74e9d301230\" (UID: \"b5080708-ea89-4bbd-a33c-d74e9d301230\") "
Nov 03 20:18:31.051099 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.051022    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5080708-ea89-4bbd-a33c-d74e9d301230-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5080708-ea89-4bbd-a33c-d74e9d301230-maas-api-tls\") pod \"b5080708-ea89-4bbd-a33c-d74e9d301230\" (UID: \"b5080708-ea89-4bbd-a33c-d74e9d301230\") "
Nov 03 20:18:31.051178 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.051133    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5080708-ea89-4bbd-a33c-d74e9d301230-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5080708-ea89-4bbd-a33c-d74e9d301230-openshift-service-ca\") pod \"b5080708-ea89-4bbd-a33c-d74e9d301230\" (UID: \"b5080708-ea89-4bbd-a33c-d74e9d301230\") "
Nov 03 20:18:31.051178 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.051168    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5080708-ea89-4bbd-a33c-d74e9d301230-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5080708-ea89-4bbd-a33c-d74e9d301230-metrics-certs\") pod \"b5080708-ea89-4bbd-a33c-d74e9d301230\" (UID: \"b5080708-ea89-4bbd-a33c-d74e9d301230\") "
Nov 03 20:18:31.051553 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.051526    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5080708-ea89-4bbd-a33c-d74e9d301230-openshift-service-ca" pod "b5080708-ea89-4bbd-a33c-d74e9d301230" (UID: "b5080708-ea89-4bbd-a33c-d74e9d301230"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:31.052096 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.052071    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5080708-ea89-4bbd-a33c-d74e9d301230-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:31.056644 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.054905    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5080708-ea89-4bbd-a33c-d74e9d301230-kube-api-access-cgghx" pod "b5080708-ea89-4bbd-a33c-d74e9d301230" (UID: "b5080708-ea89-4bbd-a33c-d74e9d301230"). InnerVolumeSpecName "kube-api-access-cgghx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:31.056644 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.055489    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5080708-ea89-4bbd-a33c-d74e9d301230-maas-api-tls" pod "b5080708-ea89-4bbd-a33c-d74e9d301230" (UID: "b5080708-ea89-4bbd-a33c-d74e9d301230"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:18:31.056644 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.056273    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5080708-ea89-4bbd-a33c-d74e9d301230-metrics-certs" pod "b5080708-ea89-4bbd-a33c-d74e9d301230" (UID: "b5080708-ea89-4bbd-a33c-d74e9d301230"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:18:31.145416 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.145384    4421 generic.go:356] "Generic (PLEG): container finished" podID="b5080708-ea89-4bbd-a33c-d74e9d301230" containerID="726a97c4135c46e615f734ed0fa8f68d228481b5595b1cec3b04cf345f0d26c8" exitCode=0
Nov 03 20:18:31.145934 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.145428    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd"
Nov 03 20:18:31.145934 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.145486    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd" event={"ID":"b5080708-ea89-4bbd-a33c-d74e9d301230","Type":"ContainerDied","Data":"726a97c4135c46e615f734ed0fa8f68d228481b5595b1cec3b04cf345f0d26c8"}
Nov 03 20:18:31.145934 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.145543    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd" event={"ID":"b5080708-ea89-4bbd-a33c-d74e9d301230","Type":"ContainerDied","Data":"d998a42cdd6f601152a33edee92f60318661473d4a4bc82e83ca3a4da301aad9"}
Nov 03 20:18:31.145934 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.145575    4421 scope.go:122] "RemoveContainer" containerID="726a97c4135c46e615f734ed0fa8f68d228481b5595b1cec3b04cf345f0d26c8"
Nov 03 20:18:31.154222 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.154128    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgghx\" (UniqueName: \"kubernetes.io/projected/b5080708-ea89-4bbd-a33c-d74e9d301230-kube-api-access-cgghx\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:31.154222 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.154180    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5080708-ea89-4bbd-a33c-d74e9d301230-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:31.154222 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.154196    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5080708-ea89-4bbd-a33c-d74e9d301230-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:31.179750 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.179725    4421 scope.go:122] "RemoveContainer" containerID="726a97c4135c46e615f734ed0fa8f68d228481b5595b1cec3b04cf345f0d26c8"
Nov 03 20:18:31.181963 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:18:31.181918    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"726a97c4135c46e615f734ed0fa8f68d228481b5595b1cec3b04cf345f0d26c8\": container with ID starting with 726a97c4135c46e615f734ed0fa8f68d228481b5595b1cec3b04cf345f0d26c8 not found: ID does not exist" containerID="726a97c4135c46e615f734ed0fa8f68d228481b5595b1cec3b04cf345f0d26c8"
Nov 03 20:18:31.182082 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.181975    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"726a97c4135c46e615f734ed0fa8f68d228481b5595b1cec3b04cf345f0d26c8"} err="failed to get container status \"726a97c4135c46e615f734ed0fa8f68d228481b5595b1cec3b04cf345f0d26c8\": rpc error: code = NotFound desc = could not find container \"726a97c4135c46e615f734ed0fa8f68d228481b5595b1cec3b04cf345f0d26c8\": container with ID starting with 726a97c4135c46e615f734ed0fa8f68d228481b5595b1cec3b04cf345f0d26c8 not found: ID does not exist"
Nov 03 20:18:31.186203 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.186181    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd"]
Nov 03 20:18:31.189291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.189265    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18ba46-68ddd7b544-5x9wd"]
Nov 03 20:18:31.406213 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.406181    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"]
Nov 03 20:18:31.407225 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.407202    4421 cpu_manager.go:414] "containerMap: removing container" podUID="cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799" containerName="payload-processing"
Nov 03 20:18:31.407285 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.407231    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799" containerName="payload-processing"
Nov 03 20:18:31.407285 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.407249    4421 cpu_manager.go:414] "containerMap: removing container" podUID="b5080708-ea89-4bbd-a33c-d74e9d301230" containerName="maas-api"
Nov 03 20:18:31.407285 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.407256    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5080708-ea89-4bbd-a33c-d74e9d301230" containerName="maas-api"
Nov 03 20:18:31.407404 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.407304    4421 cpu_manager.go:414] "containerMap: removing container" podUID="f34b3118-c425-4b46-87a8-b22d7a101c6f" containerName="payload-pre-processing"
Nov 03 20:18:31.407404 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.407312    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f34b3118-c425-4b46-87a8-b22d7a101c6f" containerName="payload-pre-processing"
Nov 03 20:18:31.407505 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.407491    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f34b3118-c425-4b46-87a8-b22d7a101c6f" containerName="payload-pre-processing"
Nov 03 20:18:31.407541 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.407518    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799" containerName="payload-processing"
Nov 03 20:18:31.407541 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.407528    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5080708-ea89-4bbd-a33c-d74e9d301230" containerName="maas-api"
Nov 03 20:18:31.414173 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.414152    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.430384 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.430331    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-33652a-openshift-default-dockercfg-2bpfb\""
Nov 03 20:18:31.439136 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.439113    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"]
Nov 03 20:18:31.460666 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.460589    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-credential-socket\") pod \"e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") " pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.460823 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.460687    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a045c5d-d469-415e-8fc1-921f2ff05253-istiod-ca-cert\") pod \"e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") " pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.460823 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.460742    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-workload-socket\") pod \"e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") " pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.460957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.460912    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-envoy\") pod \"e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") " pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.460957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.460950    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-token\") pod \"e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") " pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.461059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.460969    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-data\") pod \"e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") " pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.461113 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.461073    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf5hh\" (UniqueName: \"kubernetes.io/projected/1a045c5d-d469-415e-8fc1-921f2ff05253-kube-api-access-sf5hh\") pod \"e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") " pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.461113 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.461097    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-podinfo\") pod \"e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") " pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.461213 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.461120    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-workload-certs\") pod \"e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") " pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.572620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.567025    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-envoy\") pod \"e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") " pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.572620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.567105    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-token\") pod \"e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") " pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.572620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.567139    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-data\") pod \"e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") " pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.572620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.567203    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf5hh\" (UniqueName: \"kubernetes.io/projected/1a045c5d-d469-415e-8fc1-921f2ff05253-kube-api-access-sf5hh\") pod \"e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") " pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.572620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.567234    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-podinfo\") pod \"e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") " pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.572620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.567271    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-workload-certs\") pod \"e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") " pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.572620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.567397    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-credential-socket\") pod \"e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") " pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.572620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.567485    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a045c5d-d469-415e-8fc1-921f2ff05253-istiod-ca-cert\") pod \"e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") " pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.572620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.567555    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-workload-socket\") pod \"e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") " pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.572620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.568061    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-workload-socket\") pod \"e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") " pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.590736 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.588950    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-credential-socket\") pod \"e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") " pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.590736 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.589201    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-workload-certs\") pod \"e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") " pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.602620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.592498    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-podinfo\") pod \"e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") " pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.602620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.593070    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-data\") pod \"e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") " pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.605632 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.604045    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a045c5d-d469-415e-8fc1-921f2ff05253-istiod-ca-cert\") pod \"e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") " pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.610308 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.610283    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-envoy\") pod \"e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") " pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.632255 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.632186    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf5hh\" (UniqueName: \"kubernetes.io/projected/1a045c5d-d469-415e-8fc1-921f2ff05253-kube-api-access-sf5hh\") pod \"e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") " pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.632740 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.632717    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-token\") pod \"e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") " pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.731622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.728805    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:31.840827 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.840796    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"]
Nov 03 20:18:31.850941 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.848455    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:31.871755 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.871664    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"]
Nov 03 20:18:31.884718 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.884248    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:31.885734 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.885709    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"]
Nov 03 20:18:31.904976 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.904946    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"]
Nov 03 20:18:31.985322 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.985232    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-istio-envoy\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:31.985472 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.985337    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-workload-certs\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:31.985472 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.985425    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6feb18b2-b54f-4850-b7d0-59c872386051-istiod-ca-cert\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:31.985560 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.985492    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-istio-envoy\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:31.985560 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.985521    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acbca941-61d7-4a7a-86d9-76494049a25d-istio-token\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:31.985560 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.985542    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acbca941-61d7-4a7a-86d9-76494049a25d-istio-podinfo\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:31.985693 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.985567    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-credential-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:31.985729 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.985690    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs8fr\" (UniqueName: \"kubernetes.io/projected/6feb18b2-b54f-4850-b7d0-59c872386051-kube-api-access-zs8fr\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:31.985791 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.985776    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-istio-data\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:31.985823 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.985800    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-workload-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:31.985854 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.985824    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcj6v\" (UniqueName: \"kubernetes.io/projected/acbca941-61d7-4a7a-86d9-76494049a25d-kube-api-access-rcj6v\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:31.985892 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.985873    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-workload-certs\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:31.985929 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.985910    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6feb18b2-b54f-4850-b7d0-59c872386051-istio-podinfo\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:31.986007 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.985996    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-credential-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:31.986055 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.986044    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acbca941-61d7-4a7a-86d9-76494049a25d-istiod-ca-cert\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:31.986145 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.986124    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-istio-data\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:31.986179 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.986161    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-workload-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:31.986245 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:31.986234    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6feb18b2-b54f-4850-b7d0-59c872386051-istio-token\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:32.020263 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.020225    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5080708-ea89-4bbd-a33c-d74e9d301230" path="/var/lib/kubelet/pods/b5080708-ea89-4bbd-a33c-d74e9d301230/volumes"
Nov 03 20:18:32.020756 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.020737    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799" path="/var/lib/kubelet/pods/cf9fbd7f-d1ef-4d77-8f1d-d8c6e64cd799/volumes"
Nov 03 20:18:32.021149 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.021132    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f34b3118-c425-4b46-87a8-b22d7a101c6f" path="/var/lib/kubelet/pods/f34b3118-c425-4b46-87a8-b22d7a101c6f/volumes"
Nov 03 20:18:32.078997 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.078970    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"]
Nov 03 20:18:32.079819 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:18:32.079794    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a045c5d_d469_415e_8fc1_921f2ff05253.slice/crio-e85c264b31059723fb36d56d7efcfd52770e1bbf4f8664511732e6b7cd6f9893 WatchSource:0}: Error finding container e85c264b31059723fb36d56d7efcfd52770e1bbf4f8664511732e6b7cd6f9893: Status 404 returned error can't find the container with id e85c264b31059723fb36d56d7efcfd52770e1bbf4f8664511732e6b7cd6f9893
Nov 03 20:18:32.082770 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.082737    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:32.082879 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.082798    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:32.082879 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.082828    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:32.087579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.087549    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6feb18b2-b54f-4850-b7d0-59c872386051-istio-token\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:32.087695 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.087634    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-istio-envoy\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:32.087695 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.087672    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-workload-certs\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:32.087819 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.087730    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6feb18b2-b54f-4850-b7d0-59c872386051-istiod-ca-cert\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:32.087819 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.087761    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-istio-envoy\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:32.087819 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.087786    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acbca941-61d7-4a7a-86d9-76494049a25d-istio-token\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:32.087819 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.087813    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acbca941-61d7-4a7a-86d9-76494049a25d-istio-podinfo\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:32.088020 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.087837    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-credential-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:32.088961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.088088    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-workload-certs\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:32.088961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.088125    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-credential-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:32.088961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.088130    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs8fr\" (UniqueName: \"kubernetes.io/projected/6feb18b2-b54f-4850-b7d0-59c872386051-kube-api-access-zs8fr\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:32.088961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.088259    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-istio-data\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:32.088961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.088293    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-workload-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:32.088961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.088333    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcj6v\" (UniqueName: \"kubernetes.io/projected/acbca941-61d7-4a7a-86d9-76494049a25d-kube-api-access-rcj6v\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:32.088961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.088361    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-workload-certs\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:32.088961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.088406    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6feb18b2-b54f-4850-b7d0-59c872386051-istio-podinfo\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:32.088961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.088477    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-credential-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:32.088961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.088539    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acbca941-61d7-4a7a-86d9-76494049a25d-istiod-ca-cert\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:32.088961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.088647    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-istio-data\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:32.088961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.088663    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-istio-data\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:32.088961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.088681    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-workload-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:32.088961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.088667    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-workload-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:32.088961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.088843    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6feb18b2-b54f-4850-b7d0-59c872386051-istiod-ca-cert\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:32.088961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.088853    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-workload-certs\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:32.088961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.088906    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-credential-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:32.088961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.088941    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-workload-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:32.089967 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.089070    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-istio-data\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:32.089967 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.089453    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acbca941-61d7-4a7a-86d9-76494049a25d-istiod-ca-cert\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:32.091186 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.091143    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-istio-envoy\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:32.091186 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.091149    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-istio-envoy\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:32.091691 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.091620    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acbca941-61d7-4a7a-86d9-76494049a25d-istio-podinfo\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:32.091820 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.091807    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6feb18b2-b54f-4850-b7d0-59c872386051-istio-podinfo\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:32.101294 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.101273    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs8fr\" (UniqueName: \"kubernetes.io/projected/6feb18b2-b54f-4850-b7d0-59c872386051-kube-api-access-zs8fr\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:32.101507 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.101487    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acbca941-61d7-4a7a-86d9-76494049a25d-istio-token\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:32.101898 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.101853    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6feb18b2-b54f-4850-b7d0-59c872386051-istio-token\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:32.101992 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.101975    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcj6v\" (UniqueName: \"kubernetes.io/projected/acbca941-61d7-4a7a-86d9-76494049a25d-kube-api-access-rcj6v\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:32.187709 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.187665    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9" event={"ID":"1a045c5d-d469-415e-8fc1-921f2ff05253","Type":"ContainerStarted","Data":"e85c264b31059723fb36d56d7efcfd52770e1bbf4f8664511732e6b7cd6f9893"}
Nov 03 20:18:32.208384 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.208194    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:32.249734 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.249665    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:32.516443 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.516417    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"]
Nov 03 20:18:32.516891 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:18:32.516862    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6feb18b2_b54f_4850_b7d0_59c872386051.slice/crio-fba618d23610837198abfabf127cec3317ed99c761f1afea7eac4c23a439f8eb WatchSource:0}: Error finding container fba618d23610837198abfabf127cec3317ed99c761f1afea7eac4c23a439f8eb: Status 404 returned error can't find the container with id fba618d23610837198abfabf127cec3317ed99c761f1afea7eac4c23a439f8eb
Nov 03 20:18:32.519881 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.519843    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:32.519958 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.519932    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:32.520002 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.519981    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:32.603150 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:18:32.603110    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacbca941_61d7_4a7a_86d9_76494049a25d.slice/crio-4471ea87fb4d4023d36ef5f6449347378096257fb3ba7cb5ccb822a5a6ac58c0 WatchSource:0}: Error finding container 4471ea87fb4d4023d36ef5f6449347378096257fb3ba7cb5ccb822a5a6ac58c0: Status 404 returned error can't find the container with id 4471ea87fb4d4023d36ef5f6449347378096257fb3ba7cb5ccb822a5a6ac58c0
Nov 03 20:18:32.605109 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.605087    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"]
Nov 03 20:18:32.605695 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.605664    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:32.605800 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.605742    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:32.605863 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:32.605796    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:33.219253 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:33.219216    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl" event={"ID":"6feb18b2-b54f-4850-b7d0-59c872386051","Type":"ContainerStarted","Data":"c0e4f1e7d0eb120a5cbb9e2d9b8cbc41978bad4f48dda580751ec162337bef41"}
Nov 03 20:18:33.267624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:33.265251    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl" event={"ID":"6feb18b2-b54f-4850-b7d0-59c872386051","Type":"ContainerStarted","Data":"fba618d23610837198abfabf127cec3317ed99c761f1afea7eac4c23a439f8eb"}
Nov 03 20:18:33.267624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:33.265299    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:33.267624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:33.265324    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9" event={"ID":"1a045c5d-d469-415e-8fc1-921f2ff05253","Type":"ContainerStarted","Data":"61dfbbabb674ae282203f19adf30517d5d084158901086054ca0032998cd158f"}
Nov 03 20:18:33.267624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:33.242531    4421 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 20:18:33.267624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:33.265388    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl" podUID="6feb18b2-b54f-4850-b7d0-59c872386051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 20:18:33.267624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:33.264824    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl" podStartSLOduration=2.264802706 podStartE2EDuration="2.264802706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:33.260344817 +0000 UTC m=+1043.393473281" watchObservedRunningTime="2026-11-03 20:18:33.264802706 +0000 UTC m=+1043.397931168"
Nov 03 20:18:33.284726 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:33.277960    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6" event={"ID":"acbca941-61d7-4a7a-86d9-76494049a25d","Type":"ContainerStarted","Data":"9937db11831cd27f555bdf24afbd9a91b3056bb80b352feef547e11835b2a6b5"}
Nov 03 20:18:33.284726 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:33.278032    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6" event={"ID":"acbca941-61d7-4a7a-86d9-76494049a25d","Type":"ContainerStarted","Data":"4471ea87fb4d4023d36ef5f6449347378096257fb3ba7cb5ccb822a5a6ac58c0"}
Nov 03 20:18:33.296118 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:33.292664    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9" podStartSLOduration=2.292643268 podStartE2EDuration="2.292643268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:33.291091853 +0000 UTC m=+1043.424220318" watchObservedRunningTime="2026-11-03 20:18:33.292643268 +0000 UTC m=+1043.425771731"
Nov 03 20:18:33.320442 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:33.318176    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6" podStartSLOduration=2.318160241 podStartE2EDuration="2.318160241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:33.316481859 +0000 UTC m=+1043.449610321" watchObservedRunningTime="2026-11-03 20:18:33.318160241 +0000 UTC m=+1043.451288703"
Nov 03 20:18:33.729685 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:33.729647    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:33.735754 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:33.735729    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:34.121612 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:34.119268    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"]
Nov 03 20:18:34.121612 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:34.119497    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4" podUID="f3ad6df8-70a3-4dd1-a5e7-fc79003fb609" containerName="istio-proxy" containerID="cri-o://36d8082d42a2869522600c3c25b1b7d66ed747096e7d2171c6725857561d8189" gracePeriod=30
Nov 03 20:18:34.125898 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:34.125370    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"]
Nov 03 20:18:34.129963 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:34.129941    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"]
Nov 03 20:18:34.461377 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:34.460845    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:34.461377 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:34.460963    4421 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 20:18:34.461377 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:34.461006    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6" podUID="acbca941-61d7-4a7a-86d9-76494049a25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 20:18:34.534649 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:34.534589    4421 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 20:18:34.534826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:34.534677    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl" podUID="6feb18b2-b54f-4850-b7d0-59c872386051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 20:18:34.540504 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:34.537219    4421 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6" secret="" err="secret \"e2e-api-b-18ba46-openshift-default-dockercfg-zvvnj\" not found"
Nov 03 20:18:34.540504 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:34.537877    4421 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl" secret="" err="secret \"e2e-api-b-18ba46-openshift-default-dockercfg-zvvnj\" not found"
Nov 03 20:18:34.540504 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:34.538389    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:34.540504 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:34.540280    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:18:35.210209 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:35.210176    4421 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 20:18:35.210381 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:35.210247    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl" podUID="6feb18b2-b54f-4850-b7d0-59c872386051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 20:18:35.257278 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:35.250943    4421 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 20:18:35.257278 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:35.251015    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6" podUID="acbca941-61d7-4a7a-86d9-76494049a25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 20:18:35.581688 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:35.581564    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl" podUID="6feb18b2-b54f-4850-b7d0-59c872386051" containerName="istio-proxy" containerID="cri-o://c0e4f1e7d0eb120a5cbb9e2d9b8cbc41978bad4f48dda580751ec162337bef41" gracePeriod=30
Nov 03 20:18:35.582205 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:35.582069    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6" podUID="acbca941-61d7-4a7a-86d9-76494049a25d" containerName="istio-proxy" containerID="cri-o://9937db11831cd27f555bdf24afbd9a91b3056bb80b352feef547e11835b2a6b5" gracePeriod=30
Nov 03 20:18:37.014702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:37.014669    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5"]
Nov 03 20:18:37.028523 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:37.028309    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5"]
Nov 03 20:18:37.028523 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:37.028356    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz"]
Nov 03 20:18:37.028784 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:37.028582    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5"
Nov 03 20:18:37.033672 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:37.033649    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz"
Nov 03 20:18:37.035934 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:37.035899    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz"]
Nov 03 20:18:37.036651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:37.036631    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-33652a\""
Nov 03 20:18:37.037954 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:37.037931    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-33652a-dockercfg-kgrzp\""
Nov 03 20:18:37.110642 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:37.109491    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cac44a07-d8cf-47e4-93d0-b237eb36b446-plugins-config-volume\") pod \"payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5\" (UID: \"cac44a07-d8cf-47e4-93d0-b237eb36b446\") " pod="openshift-ingress/payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5"
Nov 03 20:18:37.110642 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:37.109615    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55a43d29-6ffd-484b-8dcc-a827785a46f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz\" (UID: \"55a43d29-6ffd-484b-8dcc-a827785a46f1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz"
Nov 03 20:18:37.110642 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:37.109735    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6wdn\" (UniqueName: \"kubernetes.io/projected/55a43d29-6ffd-484b-8dcc-a827785a46f1-kube-api-access-t6wdn\") pod \"payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz\" (UID: \"55a43d29-6ffd-484b-8dcc-a827785a46f1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz"
Nov 03 20:18:37.111169 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:37.110999    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp2pf\" (UniqueName: \"kubernetes.io/projected/cac44a07-d8cf-47e4-93d0-b237eb36b446-kube-api-access-jp2pf\") pod \"payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5\" (UID: \"cac44a07-d8cf-47e4-93d0-b237eb36b446\") " pod="openshift-ingress/payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5"
Nov 03 20:18:37.216095 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:37.216054    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp2pf\" (UniqueName: \"kubernetes.io/projected/cac44a07-d8cf-47e4-93d0-b237eb36b446-kube-api-access-jp2pf\") pod \"payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5\" (UID: \"cac44a07-d8cf-47e4-93d0-b237eb36b446\") " pod="openshift-ingress/payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5"
Nov 03 20:18:37.216424 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:37.216402    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cac44a07-d8cf-47e4-93d0-b237eb36b446-plugins-config-volume\") pod \"payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5\" (UID: \"cac44a07-d8cf-47e4-93d0-b237eb36b446\") " pod="openshift-ingress/payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5"
Nov 03 20:18:37.226935 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:37.226886    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55a43d29-6ffd-484b-8dcc-a827785a46f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz\" (UID: \"55a43d29-6ffd-484b-8dcc-a827785a46f1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz"
Nov 03 20:18:37.227425 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:37.227396    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6wdn\" (UniqueName: \"kubernetes.io/projected/55a43d29-6ffd-484b-8dcc-a827785a46f1-kube-api-access-t6wdn\") pod \"payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz\" (UID: \"55a43d29-6ffd-484b-8dcc-a827785a46f1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz"
Nov 03 20:18:37.240699 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:37.229336    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55a43d29-6ffd-484b-8dcc-a827785a46f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz\" (UID: \"55a43d29-6ffd-484b-8dcc-a827785a46f1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz"
Nov 03 20:18:37.248691 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:37.248656    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cac44a07-d8cf-47e4-93d0-b237eb36b446-plugins-config-volume\") pod \"payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5\" (UID: \"cac44a07-d8cf-47e4-93d0-b237eb36b446\") " pod="openshift-ingress/payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5"
Nov 03 20:18:37.293407 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:37.293328    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp2pf\" (UniqueName: \"kubernetes.io/projected/cac44a07-d8cf-47e4-93d0-b237eb36b446-kube-api-access-jp2pf\") pod \"payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5\" (UID: \"cac44a07-d8cf-47e4-93d0-b237eb36b446\") " pod="openshift-ingress/payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5"
Nov 03 20:18:37.321332 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:37.321299    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6wdn\" (UniqueName: \"kubernetes.io/projected/55a43d29-6ffd-484b-8dcc-a827785a46f1-kube-api-access-t6wdn\") pod \"payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz\" (UID: \"55a43d29-6ffd-484b-8dcc-a827785a46f1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz"
Nov 03 20:18:37.444136 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:37.444093    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5"
Nov 03 20:18:37.514555 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:37.444533    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz"
Nov 03 20:18:38.100394 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:38.100300    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5"]
Nov 03 20:18:38.298820 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:38.295781    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz"]
Nov 03 20:18:38.687251 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:38.685912    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz" event={"ID":"55a43d29-6ffd-484b-8dcc-a827785a46f1","Type":"ContainerStarted","Data":"919e442fcd12eacc4118fdb9afabd8b67cabd5f2b24c49fd309eb3d81ad7c8b5"}
Nov 03 20:18:38.687251 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:38.685985    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz" event={"ID":"55a43d29-6ffd-484b-8dcc-a827785a46f1","Type":"ContainerStarted","Data":"526d7ea22f7d37c3183a66f15d46df8cc468f2ca03bfafef6af8ae23af651723"}
Nov 03 20:18:38.687251 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:38.687142    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz"
Nov 03 20:18:38.695548 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:38.695520    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5" event={"ID":"cac44a07-d8cf-47e4-93d0-b237eb36b446","Type":"ContainerStarted","Data":"c27864dfd7c435e219a18a253074264e84776e29798e0ea43274121935d4af9a"}
Nov 03 20:18:38.695758 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:38.695685    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5" event={"ID":"cac44a07-d8cf-47e4-93d0-b237eb36b446","Type":"ContainerStarted","Data":"a747d1d269fa9e45b63fa38a3071746c8eb7b670c255f532f9b020af3106477b"}
Nov 03 20:18:38.695968 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:38.695953    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5"
Nov 03 20:18:38.710649 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:38.710588    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz" podStartSLOduration=2.710573565 podStartE2EDuration="2.710573565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:38.706795935 +0000 UTC m=+1048.839924398" watchObservedRunningTime="2026-11-03 20:18:38.710573565 +0000 UTC m=+1048.843702026"
Nov 03 20:18:38.728719 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:38.728677    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5" podStartSLOduration=2.7286655890000002 podStartE2EDuration="2.728665589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:38.724283763 +0000 UTC m=+1048.857412225" watchObservedRunningTime="2026-11-03 20:18:38.728665589 +0000 UTC m=+1048.861794052"
Nov 03 20:18:39.172670 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.162693    4421 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": read tcp 10.134.0.2:38170->10.134.0.89:15021: read: connection reset by peer" start-of-body=
Nov 03 20:18:39.172670 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.162750    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4" podUID="f3ad6df8-70a3-4dd1-a5e7-fc79003fb609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": read tcp 10.134.0.2:38170->10.134.0.89:15021: read: connection reset by peer"
Nov 03 20:18:39.574608 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.574569    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:39.704488 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.704455    4421 generic.go:356] "Generic (PLEG): container finished" podID="f3ad6df8-70a3-4dd1-a5e7-fc79003fb609" containerID="36d8082d42a2869522600c3c25b1b7d66ed747096e7d2171c6725857561d8189" exitCode=0
Nov 03 20:18:39.704703 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.704510    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"
Nov 03 20:18:39.704703 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.704519    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4" event={"ID":"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609","Type":"ContainerDied","Data":"36d8082d42a2869522600c3c25b1b7d66ed747096e7d2171c6725857561d8189"}
Nov 03 20:18:39.704703 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.704591    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4" event={"ID":"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609","Type":"ContainerDied","Data":"2500cb78568060a8615177bdc66ff2feddf5e8b9c1fda985507bba3aec6d830d"}
Nov 03 20:18:39.704703 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.704673    4421 scope.go:122] "RemoveContainer" containerID="36d8082d42a2869522600c3c25b1b7d66ed747096e7d2171c6725857561d8189"
Nov 03 20:18:39.715822 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.715802    4421 scope.go:122] "RemoveContainer" containerID="36d8082d42a2869522600c3c25b1b7d66ed747096e7d2171c6725857561d8189"
Nov 03 20:18:39.716179 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:18:39.716153    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d8082d42a2869522600c3c25b1b7d66ed747096e7d2171c6725857561d8189\": container with ID starting with 36d8082d42a2869522600c3c25b1b7d66ed747096e7d2171c6725857561d8189 not found: ID does not exist" containerID="36d8082d42a2869522600c3c25b1b7d66ed747096e7d2171c6725857561d8189"
Nov 03 20:18:39.716323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.716204    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d8082d42a2869522600c3c25b1b7d66ed747096e7d2171c6725857561d8189"} err="failed to get container status \"36d8082d42a2869522600c3c25b1b7d66ed747096e7d2171c6725857561d8189\": rpc error: code = NotFound desc = could not find container \"36d8082d42a2869522600c3c25b1b7d66ed747096e7d2171c6725857561d8189\": container with ID starting with 36d8082d42a2869522600c3c25b1b7d66ed747096e7d2171c6725857561d8189 not found: ID does not exist"
Nov 03 20:18:39.717436 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.717411    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-token\" (UniqueName: \"kubernetes.io/projected/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-token\") pod \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") "
Nov 03 20:18:39.717568 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.717551    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-workload-certs\") pod \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") "
Nov 03 20:18:39.717656 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.717640    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-workload-socket\") pod \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") "
Nov 03 20:18:39.717750 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.717731    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-credential-socket\") pod \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") "
Nov 03 20:18:39.717873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.717856    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istiod-ca-cert\") pod \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") "
Nov 03 20:18:39.717940 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.717902    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-workload-certs" pod "f3ad6df8-70a3-4dd1-a5e7-fc79003fb609" (UID: "f3ad6df8-70a3-4dd1-a5e7-fc79003fb609"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:39.717997 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.717941    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-podinfo\") pod \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") "
Nov 03 20:18:39.717997 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.717984    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-workload-socket" pod "f3ad6df8-70a3-4dd1-a5e7-fc79003fb609" (UID: "f3ad6df8-70a3-4dd1-a5e7-fc79003fb609"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:39.718103 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.717990    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-kube-api-access-5bx5j\" (UniqueName: \"kubernetes.io/projected/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-kube-api-access-5bx5j\") pod \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") "
Nov 03 20:18:39.718103 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.718036    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-data\") pod \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") "
Nov 03 20:18:39.718215 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.718132    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-envoy\") pod \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\" (UID: \"f3ad6df8-70a3-4dd1-a5e7-fc79003fb609\") "
Nov 03 20:18:39.718505 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.718483    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-credential-socket" pod "f3ad6df8-70a3-4dd1-a5e7-fc79003fb609" (UID: "f3ad6df8-70a3-4dd1-a5e7-fc79003fb609"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:39.718659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.718635    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istiod-ca-cert" pod "f3ad6df8-70a3-4dd1-a5e7-fc79003fb609" (UID: "f3ad6df8-70a3-4dd1-a5e7-fc79003fb609"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:39.718822 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.718797    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-data" pod "f3ad6df8-70a3-4dd1-a5e7-fc79003fb609" (UID: "f3ad6df8-70a3-4dd1-a5e7-fc79003fb609"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:39.719165 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.719145    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:39.719262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.719182    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:39.719262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.719200    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:39.719262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.719214    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:39.719262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.719226    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:39.721184 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.721158    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-kube-api-access-5bx5j" pod "f3ad6df8-70a3-4dd1-a5e7-fc79003fb609" (UID: "f3ad6df8-70a3-4dd1-a5e7-fc79003fb609"). InnerVolumeSpecName "kube-api-access-5bx5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:39.721285 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.721178    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-token" pod "f3ad6df8-70a3-4dd1-a5e7-fc79003fb609" (UID: "f3ad6df8-70a3-4dd1-a5e7-fc79003fb609"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:39.721476 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.721400    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-envoy" pod "f3ad6df8-70a3-4dd1-a5e7-fc79003fb609" (UID: "f3ad6df8-70a3-4dd1-a5e7-fc79003fb609"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:39.722797 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.722775    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-podinfo" pod "f3ad6df8-70a3-4dd1-a5e7-fc79003fb609" (UID: "f3ad6df8-70a3-4dd1-a5e7-fc79003fb609"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:18:39.820488 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.820409    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:39.820488 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.820432    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:39.820488 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.820442    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bx5j\" (UniqueName: \"kubernetes.io/projected/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-kube-api-access-5bx5j\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:39.820488 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:39.820456    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:40.046710 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:40.046676    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"]
Nov 03 20:18:40.068096 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:40.068036    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-69qg4"]
Nov 03 20:18:42.018571 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:42.018539    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3ad6df8-70a3-4dd1-a5e7-fc79003fb609" path="/var/lib/kubelet/pods/f3ad6df8-70a3-4dd1-a5e7-fc79003fb609/volumes"
Nov 03 20:18:42.638033 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:42.637840    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck"]
Nov 03 20:18:42.639953 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:42.639927    4421 cpu_manager.go:414] "containerMap: removing container" podUID="f3ad6df8-70a3-4dd1-a5e7-fc79003fb609" containerName="istio-proxy"
Nov 03 20:18:42.639953 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:42.639953    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3ad6df8-70a3-4dd1-a5e7-fc79003fb609" containerName="istio-proxy"
Nov 03 20:18:42.640131 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:42.640114    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3ad6df8-70a3-4dd1-a5e7-fc79003fb609" containerName="istio-proxy"
Nov 03 20:18:42.646662 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:42.646637    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck"
Nov 03 20:18:42.658166 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:42.658145    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-33652a-serving-cert\""
Nov 03 20:18:42.664782 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:42.664755    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck"]
Nov 03 20:18:42.701570 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:42.701539    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt9pm\" (UniqueName: \"kubernetes.io/projected/aad3c734-b6ba-42f2-a3f2-6794d4e86793-kube-api-access-lt9pm\") pod \"maas-api-e2e-api-b-33652a-976894fcb-j9pck\" (UID: \"aad3c734-b6ba-42f2-a3f2-6794d4e86793\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck"
Nov 03 20:18:42.701727 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:42.701704    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aad3c734-b6ba-42f2-a3f2-6794d4e86793-metrics-certs\") pod \"maas-api-e2e-api-b-33652a-976894fcb-j9pck\" (UID: \"aad3c734-b6ba-42f2-a3f2-6794d4e86793\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck"
Nov 03 20:18:42.702015 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:42.701993    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aad3c734-b6ba-42f2-a3f2-6794d4e86793-openshift-service-ca\") pod \"maas-api-e2e-api-b-33652a-976894fcb-j9pck\" (UID: \"aad3c734-b6ba-42f2-a3f2-6794d4e86793\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck"
Nov 03 20:18:42.702092 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:42.702058    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aad3c734-b6ba-42f2-a3f2-6794d4e86793-maas-api-tls\") pod \"maas-api-e2e-api-b-33652a-976894fcb-j9pck\" (UID: \"aad3c734-b6ba-42f2-a3f2-6794d4e86793\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck"
Nov 03 20:18:42.810619 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:42.808110    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aad3c734-b6ba-42f2-a3f2-6794d4e86793-metrics-certs\") pod \"maas-api-e2e-api-b-33652a-976894fcb-j9pck\" (UID: \"aad3c734-b6ba-42f2-a3f2-6794d4e86793\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck"
Nov 03 20:18:42.810619 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:42.808402    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aad3c734-b6ba-42f2-a3f2-6794d4e86793-openshift-service-ca\") pod \"maas-api-e2e-api-b-33652a-976894fcb-j9pck\" (UID: \"aad3c734-b6ba-42f2-a3f2-6794d4e86793\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck"
Nov 03 20:18:42.810619 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:42.808466    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aad3c734-b6ba-42f2-a3f2-6794d4e86793-maas-api-tls\") pod \"maas-api-e2e-api-b-33652a-976894fcb-j9pck\" (UID: \"aad3c734-b6ba-42f2-a3f2-6794d4e86793\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck"
Nov 03 20:18:42.810619 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:42.808505    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt9pm\" (UniqueName: \"kubernetes.io/projected/aad3c734-b6ba-42f2-a3f2-6794d4e86793-kube-api-access-lt9pm\") pod \"maas-api-e2e-api-b-33652a-976894fcb-j9pck\" (UID: \"aad3c734-b6ba-42f2-a3f2-6794d4e86793\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck"
Nov 03 20:18:42.816587 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:42.816552    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aad3c734-b6ba-42f2-a3f2-6794d4e86793-metrics-certs\") pod \"maas-api-e2e-api-b-33652a-976894fcb-j9pck\" (UID: \"aad3c734-b6ba-42f2-a3f2-6794d4e86793\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck"
Nov 03 20:18:42.817304 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:42.817276    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aad3c734-b6ba-42f2-a3f2-6794d4e86793-openshift-service-ca\") pod \"maas-api-e2e-api-b-33652a-976894fcb-j9pck\" (UID: \"aad3c734-b6ba-42f2-a3f2-6794d4e86793\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck"
Nov 03 20:18:42.862261 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:42.858773    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aad3c734-b6ba-42f2-a3f2-6794d4e86793-maas-api-tls\") pod \"maas-api-e2e-api-b-33652a-976894fcb-j9pck\" (UID: \"aad3c734-b6ba-42f2-a3f2-6794d4e86793\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck"
Nov 03 20:18:42.866986 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:42.866939    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt9pm\" (UniqueName: \"kubernetes.io/projected/aad3c734-b6ba-42f2-a3f2-6794d4e86793-kube-api-access-lt9pm\") pod \"maas-api-e2e-api-b-33652a-976894fcb-j9pck\" (UID: \"aad3c734-b6ba-42f2-a3f2-6794d4e86793\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck"
Nov 03 20:18:42.975704 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:42.975670    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck"
Nov 03 20:18:43.381748 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:43.381722    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck"]
Nov 03 20:18:43.382397 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:18:43.382333    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad3c734_b6ba_42f2_a3f2_6794d4e86793.slice/crio-264b6f444a93faeb3664ca77b866e3e16fb4a293c0dba762e4f7a2c2b40a1eed WatchSource:0}: Error finding container 264b6f444a93faeb3664ca77b866e3e16fb4a293c0dba762e4f7a2c2b40a1eed: Status 404 returned error can't find the container with id 264b6f444a93faeb3664ca77b866e3e16fb4a293c0dba762e4f7a2c2b40a1eed
Nov 03 20:18:43.879422 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:43.879385    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck" event={"ID":"aad3c734-b6ba-42f2-a3f2-6794d4e86793","Type":"ContainerStarted","Data":"264b6f444a93faeb3664ca77b866e3e16fb4a293c0dba762e4f7a2c2b40a1eed"}
Nov 03 20:18:44.518199 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.518160    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"]
Nov 03 20:18:44.523904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.523879    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.536961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.536934    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"]
Nov 03 20:18:44.608307 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.608274    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.608466 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.608443    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2982ead4-1f06-4ce4-af76-81e660252c20-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.608548 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.608503    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.608658 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.608563    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx6pz\" (UniqueName: \"kubernetes.io/projected/2982ead4-1f06-4ce4-af76-81e660252c20-kube-api-access-dx6pz\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.608658 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.608625    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2982ead4-1f06-4ce4-af76-81e660252c20-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.608764 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.608700    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.608764 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.608747    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.608878 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.608831    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.608878 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.608868    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2982ead4-1f06-4ce4-af76-81e660252c20-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.710137 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.710104    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.710309 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.710172    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.710309 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.710212    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.710309 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.710260    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2982ead4-1f06-4ce4-af76-81e660252c20-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.710445 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.710342    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.710501 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.710476    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2982ead4-1f06-4ce4-af76-81e660252c20-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.710561 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.710544    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.710874 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.710579    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx6pz\" (UniqueName: \"kubernetes.io/projected/2982ead4-1f06-4ce4-af76-81e660252c20-kube-api-access-dx6pz\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.710874 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.710638    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2982ead4-1f06-4ce4-af76-81e660252c20-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.710874 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.710638    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.710874 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.710767    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.711281 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.711263    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2982ead4-1f06-4ce4-af76-81e660252c20-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.711456 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.711433    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.714448 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.714104    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.744274 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.728042    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2982ead4-1f06-4ce4-af76-81e660252c20-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.744274 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.728045    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.745200 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.745173    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2982ead4-1f06-4ce4-af76-81e660252c20-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.745327 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.745279    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx6pz\" (UniqueName: \"kubernetes.io/projected/2982ead4-1f06-4ce4-af76-81e660252c20-kube-api-access-dx6pz\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.816198 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.816173    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:44.846750 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.846720    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:44.897231 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.896147    4421 generic.go:356] "Generic (PLEG): container finished" podID="acbca941-61d7-4a7a-86d9-76494049a25d" containerID="9937db11831cd27f555bdf24afbd9a91b3056bb80b352feef547e11835b2a6b5" exitCode=0
Nov 03 20:18:44.897231 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.896258    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6" event={"ID":"acbca941-61d7-4a7a-86d9-76494049a25d","Type":"ContainerDied","Data":"9937db11831cd27f555bdf24afbd9a91b3056bb80b352feef547e11835b2a6b5"}
Nov 03 20:18:44.899005 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.898811    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck" event={"ID":"aad3c734-b6ba-42f2-a3f2-6794d4e86793","Type":"ContainerStarted","Data":"a26cd0f3349a7315f762d4a7933677b89ab2bbf45bf205fd958de6151ed157cf"}
Nov 03 20:18:44.906191 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.900773    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck"
Nov 03 20:18:44.906191 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.902349    4421 generic.go:356] "Generic (PLEG): container finished" podID="6feb18b2-b54f-4850-b7d0-59c872386051" containerID="c0e4f1e7d0eb120a5cbb9e2d9b8cbc41978bad4f48dda580751ec162337bef41" exitCode=0
Nov 03 20:18:44.906191 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.902408    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl" event={"ID":"6feb18b2-b54f-4850-b7d0-59c872386051","Type":"ContainerDied","Data":"c0e4f1e7d0eb120a5cbb9e2d9b8cbc41978bad4f48dda580751ec162337bef41"}
Nov 03 20:18:44.906191 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.902442    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl" event={"ID":"6feb18b2-b54f-4850-b7d0-59c872386051","Type":"ContainerDied","Data":"fba618d23610837198abfabf127cec3317ed99c761f1afea7eac4c23a439f8eb"}
Nov 03 20:18:44.906191 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.902506    4421 scope.go:122] "RemoveContainer" containerID="c0e4f1e7d0eb120a5cbb9e2d9b8cbc41978bad4f48dda580751ec162337bef41"
Nov 03 20:18:44.906191 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.902686    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"
Nov 03 20:18:44.917654 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.914042    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-workload-certs\") pod \"6feb18b2-b54f-4850-b7d0-59c872386051\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") "
Nov 03 20:18:44.917654 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.914123    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6feb18b2-b54f-4850-b7d0-59c872386051-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6feb18b2-b54f-4850-b7d0-59c872386051-istiod-ca-cert\") pod \"6feb18b2-b54f-4850-b7d0-59c872386051\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") "
Nov 03 20:18:44.917654 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.914274    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-istio-data\") pod \"6feb18b2-b54f-4850-b7d0-59c872386051\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") "
Nov 03 20:18:44.917654 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.914381    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-istio-envoy\") pod \"6feb18b2-b54f-4850-b7d0-59c872386051\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") "
Nov 03 20:18:44.917654 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.914474    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6feb18b2-b54f-4850-b7d0-59c872386051-kube-api-access-zs8fr\" (UniqueName: \"kubernetes.io/projected/6feb18b2-b54f-4850-b7d0-59c872386051-kube-api-access-zs8fr\") pod \"6feb18b2-b54f-4850-b7d0-59c872386051\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") "
Nov 03 20:18:44.917654 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.914509    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-workload-socket\") pod \"6feb18b2-b54f-4850-b7d0-59c872386051\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") "
Nov 03 20:18:44.917654 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.914555    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-credential-socket\") pod \"6feb18b2-b54f-4850-b7d0-59c872386051\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") "
Nov 03 20:18:44.917654 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.914630    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6feb18b2-b54f-4850-b7d0-59c872386051-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6feb18b2-b54f-4850-b7d0-59c872386051-istio-podinfo\") pod \"6feb18b2-b54f-4850-b7d0-59c872386051\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") "
Nov 03 20:18:44.917654 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.914696    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6feb18b2-b54f-4850-b7d0-59c872386051-istio-token\" (UniqueName: \"kubernetes.io/projected/6feb18b2-b54f-4850-b7d0-59c872386051-istio-token\") pod \"6feb18b2-b54f-4850-b7d0-59c872386051\" (UID: \"6feb18b2-b54f-4850-b7d0-59c872386051\") "
Nov 03 20:18:44.923370 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.922765    4421 scope.go:122] "RemoveContainer" containerID="c0e4f1e7d0eb120a5cbb9e2d9b8cbc41978bad4f48dda580751ec162337bef41"
Nov 03 20:18:44.923370 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.923206    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-istio-data" pod "6feb18b2-b54f-4850-b7d0-59c872386051" (UID: "6feb18b2-b54f-4850-b7d0-59c872386051"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:44.923370 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.923233    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-workload-socket" pod "6feb18b2-b54f-4850-b7d0-59c872386051" (UID: "6feb18b2-b54f-4850-b7d0-59c872386051"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:44.923370 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:18:44.923305    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0e4f1e7d0eb120a5cbb9e2d9b8cbc41978bad4f48dda580751ec162337bef41\": container with ID starting with c0e4f1e7d0eb120a5cbb9e2d9b8cbc41978bad4f48dda580751ec162337bef41 not found: ID does not exist" containerID="c0e4f1e7d0eb120a5cbb9e2d9b8cbc41978bad4f48dda580751ec162337bef41"
Nov 03 20:18:44.923370 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.923337    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0e4f1e7d0eb120a5cbb9e2d9b8cbc41978bad4f48dda580751ec162337bef41"} err="failed to get container status \"c0e4f1e7d0eb120a5cbb9e2d9b8cbc41978bad4f48dda580751ec162337bef41\": rpc error: code = NotFound desc = could not find container \"c0e4f1e7d0eb120a5cbb9e2d9b8cbc41978bad4f48dda580751ec162337bef41\": container with ID starting with c0e4f1e7d0eb120a5cbb9e2d9b8cbc41978bad4f48dda580751ec162337bef41 not found: ID does not exist"
Nov 03 20:18:44.923687 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.923544    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-workload-certs" pod "6feb18b2-b54f-4850-b7d0-59c872386051" (UID: "6feb18b2-b54f-4850-b7d0-59c872386051"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:44.923687 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.923558    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-credential-socket" pod "6feb18b2-b54f-4850-b7d0-59c872386051" (UID: "6feb18b2-b54f-4850-b7d0-59c872386051"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:44.929956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.924079    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6feb18b2-b54f-4850-b7d0-59c872386051-istiod-ca-cert" pod "6feb18b2-b54f-4850-b7d0-59c872386051" (UID: "6feb18b2-b54f-4850-b7d0-59c872386051"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:44.929956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.925917    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck" podStartSLOduration=2.471103042 podStartE2EDuration="2.925900221s" totalImagesPullingTime="454.797179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:44.924215346 +0000 UTC m=+1055.057343808" watchObservedRunningTime="2026-11-03 20:18:44.925900221 +0000 UTC m=+1055.059028698"
Nov 03 20:18:44.934054 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.934016    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6feb18b2-b54f-4850-b7d0-59c872386051-istio-token" pod "6feb18b2-b54f-4850-b7d0-59c872386051" (UID: "6feb18b2-b54f-4850-b7d0-59c872386051"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:44.934305 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.934116    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-istio-envoy" pod "6feb18b2-b54f-4850-b7d0-59c872386051" (UID: "6feb18b2-b54f-4850-b7d0-59c872386051"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:44.954619 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.954041    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6feb18b2-b54f-4850-b7d0-59c872386051-kube-api-access-zs8fr" pod "6feb18b2-b54f-4850-b7d0-59c872386051" (UID: "6feb18b2-b54f-4850-b7d0-59c872386051"). InnerVolumeSpecName "kube-api-access-zs8fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:44.957501 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:44.957459    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6feb18b2-b54f-4850-b7d0-59c872386051-istio-podinfo" pod "6feb18b2-b54f-4850-b7d0-59c872386051" (UID: "6feb18b2-b54f-4850-b7d0-59c872386051"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:18:45.017113 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.017084    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:45.017113 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.017112    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:45.017303 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.017127    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs8fr\" (UniqueName: \"kubernetes.io/projected/6feb18b2-b54f-4850-b7d0-59c872386051-kube-api-access-zs8fr\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:45.017303 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.017140    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:45.017303 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.017153    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:45.017303 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.017166    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6feb18b2-b54f-4850-b7d0-59c872386051-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:45.017303 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.017179    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6feb18b2-b54f-4850-b7d0-59c872386051-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:45.017303 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.017192    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6feb18b2-b54f-4850-b7d0-59c872386051-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:45.017303 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.017204    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6feb18b2-b54f-4850-b7d0-59c872386051-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:45.026869 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.026825    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:45.133826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.118138    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-istio-data\") pod \"acbca941-61d7-4a7a-86d9-76494049a25d\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") "
Nov 03 20:18:45.133826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.118236    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acbca941-61d7-4a7a-86d9-76494049a25d-istio-token\" (UniqueName: \"kubernetes.io/projected/acbca941-61d7-4a7a-86d9-76494049a25d-istio-token\") pod \"acbca941-61d7-4a7a-86d9-76494049a25d\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") "
Nov 03 20:18:45.133826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.118265    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acbca941-61d7-4a7a-86d9-76494049a25d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acbca941-61d7-4a7a-86d9-76494049a25d-istiod-ca-cert\") pod \"acbca941-61d7-4a7a-86d9-76494049a25d\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") "
Nov 03 20:18:45.133826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.118391    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-istio-envoy\") pod \"acbca941-61d7-4a7a-86d9-76494049a25d\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") "
Nov 03 20:18:45.133826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.118482    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-credential-socket\") pod \"acbca941-61d7-4a7a-86d9-76494049a25d\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") "
Nov 03 20:18:45.133826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.118553    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-workload-certs\") pod \"acbca941-61d7-4a7a-86d9-76494049a25d\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") "
Nov 03 20:18:45.133826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.118690    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-workload-socket\") pod \"acbca941-61d7-4a7a-86d9-76494049a25d\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") "
Nov 03 20:18:45.133826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.118741    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acbca941-61d7-4a7a-86d9-76494049a25d-kube-api-access-rcj6v\" (UniqueName: \"kubernetes.io/projected/acbca941-61d7-4a7a-86d9-76494049a25d-kube-api-access-rcj6v\") pod \"acbca941-61d7-4a7a-86d9-76494049a25d\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") "
Nov 03 20:18:45.133826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.118834    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/acbca941-61d7-4a7a-86d9-76494049a25d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acbca941-61d7-4a7a-86d9-76494049a25d-istio-podinfo\") pod \"acbca941-61d7-4a7a-86d9-76494049a25d\" (UID: \"acbca941-61d7-4a7a-86d9-76494049a25d\") "
Nov 03 20:18:45.133826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.126777    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-istio-data" pod "acbca941-61d7-4a7a-86d9-76494049a25d" (UID: "acbca941-61d7-4a7a-86d9-76494049a25d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:45.133826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.127427    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-workload-certs" pod "acbca941-61d7-4a7a-86d9-76494049a25d" (UID: "acbca941-61d7-4a7a-86d9-76494049a25d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:45.133826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.127748    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acbca941-61d7-4a7a-86d9-76494049a25d-istiod-ca-cert" pod "acbca941-61d7-4a7a-86d9-76494049a25d" (UID: "acbca941-61d7-4a7a-86d9-76494049a25d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:45.133826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.127813    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/acbca941-61d7-4a7a-86d9-76494049a25d-istio-podinfo" pod "acbca941-61d7-4a7a-86d9-76494049a25d" (UID: "acbca941-61d7-4a7a-86d9-76494049a25d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:18:45.133826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.128040    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-credential-socket" pod "acbca941-61d7-4a7a-86d9-76494049a25d" (UID: "acbca941-61d7-4a7a-86d9-76494049a25d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:45.133826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.128203    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-workload-socket" pod "acbca941-61d7-4a7a-86d9-76494049a25d" (UID: "acbca941-61d7-4a7a-86d9-76494049a25d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:45.133826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.130065    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acbca941-61d7-4a7a-86d9-76494049a25d-istio-token" pod "acbca941-61d7-4a7a-86d9-76494049a25d" (UID: "acbca941-61d7-4a7a-86d9-76494049a25d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:45.133826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.130099    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acbca941-61d7-4a7a-86d9-76494049a25d-kube-api-access-rcj6v" pod "acbca941-61d7-4a7a-86d9-76494049a25d" (UID: "acbca941-61d7-4a7a-86d9-76494049a25d"). InnerVolumeSpecName "kube-api-access-rcj6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:45.133826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.131321    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-istio-envoy" pod "acbca941-61d7-4a7a-86d9-76494049a25d" (UID: "acbca941-61d7-4a7a-86d9-76494049a25d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:45.179741 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.179712    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"]
Nov 03 20:18:45.180142 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:18:45.180120    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2982ead4_1f06_4ce4_af76_81e660252c20.slice/crio-2319eed735a3107a684ed15ea5b2c9e45e11137e40c2348c386384bdb7fe26ed WatchSource:0}: Error finding container 2319eed735a3107a684ed15ea5b2c9e45e11137e40c2348c386384bdb7fe26ed: Status 404 returned error can't find the container with id 2319eed735a3107a684ed15ea5b2c9e45e11137e40c2348c386384bdb7fe26ed
Nov 03 20:18:45.182771 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.182741    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:45.182868 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.182805    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:45.182868 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.182838    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:45.220856 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.220833    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:45.220856 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.220857    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcj6v\" (UniqueName: \"kubernetes.io/projected/acbca941-61d7-4a7a-86d9-76494049a25d-kube-api-access-rcj6v\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:45.221007 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.220867    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acbca941-61d7-4a7a-86d9-76494049a25d-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:45.221007 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.220877    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:45.221007 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.220886    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acbca941-61d7-4a7a-86d9-76494049a25d-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:45.221007 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.220894    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acbca941-61d7-4a7a-86d9-76494049a25d-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:45.221007 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.220902    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:45.221007 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.220913    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:45.221007 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.220930    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acbca941-61d7-4a7a-86d9-76494049a25d-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:45.248056 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.248007    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"]
Nov 03 20:18:45.251950 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.251928    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-4nfjl"]
Nov 03 20:18:45.913303 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.913263    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6" event={"ID":"acbca941-61d7-4a7a-86d9-76494049a25d","Type":"ContainerDied","Data":"4471ea87fb4d4023d36ef5f6449347378096257fb3ba7cb5ccb822a5a6ac58c0"}
Nov 03 20:18:45.913887 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.913335    4421 scope.go:122] "RemoveContainer" containerID="9937db11831cd27f555bdf24afbd9a91b3056bb80b352feef547e11835b2a6b5"
Nov 03 20:18:45.913887 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.913450    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"
Nov 03 20:18:45.926632 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.926580    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m" event={"ID":"2982ead4-1f06-4ce4-af76-81e660252c20","Type":"ContainerStarted","Data":"f17136f14f8b01b334fef683cecfcc87229538154de7a09e459c547d72cd3c02"}
Nov 03 20:18:45.926773 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.926660    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m" event={"ID":"2982ead4-1f06-4ce4-af76-81e660252c20","Type":"ContainerStarted","Data":"2319eed735a3107a684ed15ea5b2c9e45e11137e40c2348c386384bdb7fe26ed"}
Nov 03 20:18:45.989171 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.989122    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m" podStartSLOduration=1.989104502 podStartE2EDuration="1.989104502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:45.971983209 +0000 UTC m=+1056.105111673" watchObservedRunningTime="2026-11-03 20:18:45.989104502 +0000 UTC m=+1056.122232965"
Nov 03 20:18:45.991542 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.991504    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"]
Nov 03 20:18:45.994261 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:45.994236    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-mqgx6"]
Nov 03 20:18:46.076681 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:46.076646    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6feb18b2-b54f-4850-b7d0-59c872386051" path="/var/lib/kubelet/pods/6feb18b2-b54f-4850-b7d0-59c872386051/volumes"
Nov 03 20:18:46.077782 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:46.077695    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acbca941-61d7-4a7a-86d9-76494049a25d" path="/var/lib/kubelet/pods/acbca941-61d7-4a7a-86d9-76494049a25d/volumes"
Nov 03 20:18:46.865804 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:46.865768    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:46.867586 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:46.867555    4421 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 20:18:46.867736 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:46.867648    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m" podUID="2982ead4-1f06-4ce4-af76-81e660252c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 20:18:47.850682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:47.849000    4421 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 20:18:47.850682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:47.849067    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m" podUID="2982ead4-1f06-4ce4-af76-81e660252c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 20:18:48.861059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:48.861023    4421 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 20:18:48.861703 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:48.861674    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m" podUID="2982ead4-1f06-4ce4-af76-81e660252c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 20:18:49.715810 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:18:49.715729    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:18:49.734115 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:49.732830    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5"
Nov 03 20:18:49.863623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:49.847982    4421 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 20:18:49.863623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:49.848055    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m" podUID="2982ead4-1f06-4ce4-af76-81e660252c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 20:18:50.763630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:50.759835    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz"
Nov 03 20:18:50.851079 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:50.851042    4421 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 20:18:50.851403 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:50.851370    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m" podUID="2982ead4-1f06-4ce4-af76-81e660252c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 20:18:51.847224 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:51.847186    4421 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 20:18:51.847627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:51.847247    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m" podUID="2982ead4-1f06-4ce4-af76-81e660252c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 20:18:52.032949 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:52.032921    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck"
Nov 03 20:18:53.095836 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:53.095802    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:53.096330 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:53.096295    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:18:53.097294 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:18:53.097267    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:19:07.589866 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:07.589834    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l"]
Nov 03 20:19:11.881155 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:11.881121    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d"]
Nov 03 20:19:11.881995 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:11.881935    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d" podUID="264eaf78-ec83-4872-8a0b-e1de8f8e5a0b" containerName="maas-api" containerID="cri-o://9c2b57a363542bc4542e31143ff641adbc3ee536f68b7fef147ed3667c7ece35" gracePeriod=30
Nov 03 20:19:11.896498 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:11.896459    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d" podUID="264eaf78-ec83-4872-8a0b-e1de8f8e5a0b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.91:8443/health\": dial tcp 10.134.0.91:8443: connect: connection refused"
Nov 03 20:19:12.468894 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:12.468861    4421 generic.go:356] "Generic (PLEG): container finished" podID="264eaf78-ec83-4872-8a0b-e1de8f8e5a0b" containerID="9c2b57a363542bc4542e31143ff641adbc3ee536f68b7fef147ed3667c7ece35" exitCode=0
Nov 03 20:19:12.469087 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:12.468932    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d" event={"ID":"264eaf78-ec83-4872-8a0b-e1de8f8e5a0b","Type":"ContainerDied","Data":"9c2b57a363542bc4542e31143ff641adbc3ee536f68b7fef147ed3667c7ece35"}
Nov 03 20:19:12.496862 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:12.496833    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d"
Nov 03 20:19:12.558909 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:12.558876    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-openshift-service-ca\") pod \"264eaf78-ec83-4872-8a0b-e1de8f8e5a0b\" (UID: \"264eaf78-ec83-4872-8a0b-e1de8f8e5a0b\") "
Nov 03 20:19:12.559129 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:12.559104    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-metrics-certs\") pod \"264eaf78-ec83-4872-8a0b-e1de8f8e5a0b\" (UID: \"264eaf78-ec83-4872-8a0b-e1de8f8e5a0b\") "
Nov 03 20:19:12.559207 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:12.559193    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-maas-api-tls\") pod \"264eaf78-ec83-4872-8a0b-e1de8f8e5a0b\" (UID: \"264eaf78-ec83-4872-8a0b-e1de8f8e5a0b\") "
Nov 03 20:19:12.559331 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:12.559301    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-openshift-service-ca" pod "264eaf78-ec83-4872-8a0b-e1de8f8e5a0b" (UID: "264eaf78-ec83-4872-8a0b-e1de8f8e5a0b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:12.559415 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:12.559328    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-kube-api-access-bt6nb\" (UniqueName: \"kubernetes.io/projected/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-kube-api-access-bt6nb\") pod \"264eaf78-ec83-4872-8a0b-e1de8f8e5a0b\" (UID: \"264eaf78-ec83-4872-8a0b-e1de8f8e5a0b\") "
Nov 03 20:19:12.560137 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:12.560103    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:12.562418 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:12.562324    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-maas-api-tls" pod "264eaf78-ec83-4872-8a0b-e1de8f8e5a0b" (UID: "264eaf78-ec83-4872-8a0b-e1de8f8e5a0b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:19:12.562418 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:12.562380    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-metrics-certs" pod "264eaf78-ec83-4872-8a0b-e1de8f8e5a0b" (UID: "264eaf78-ec83-4872-8a0b-e1de8f8e5a0b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:19:12.562873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:12.562837    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-kube-api-access-bt6nb" pod "264eaf78-ec83-4872-8a0b-e1de8f8e5a0b" (UID: "264eaf78-ec83-4872-8a0b-e1de8f8e5a0b"). InnerVolumeSpecName "kube-api-access-bt6nb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:12.590497 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:12.590464    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l" podUID="f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be" containerName="payload-processing" containerID="cri-o://21ac8bc4582349cd23f23d4956eda0a0a1279dc23cc879b0f744ae04cb20999a" gracePeriod=25
Nov 03 20:19:12.661257 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:12.661224    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:12.661257 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:12.661255    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:12.661482 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:12.661266    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt6nb\" (UniqueName: \"kubernetes.io/projected/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b-kube-api-access-bt6nb\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:12.854005 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:12.853970    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l"
Nov 03 20:19:12.863976 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:12.863948    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be-plugins-config-volume\") pod \"f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be\" (UID: \"f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be\") "
Nov 03 20:19:12.864153 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:12.864140    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be-kube-api-access-fs2ls\" (UniqueName: \"kubernetes.io/projected/f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be-kube-api-access-fs2ls\") pod \"f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be\" (UID: \"f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be\") "
Nov 03 20:19:12.864281 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:12.864259    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be-plugins-config-volume" pod "f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be" (UID: "f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:12.864614 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:12.864582    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:12.866311 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:12.866285    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be-kube-api-access-fs2ls" pod "f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be" (UID: "f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be"). InnerVolumeSpecName "kube-api-access-fs2ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:12.966768 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:12.966737    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs2ls\" (UniqueName: \"kubernetes.io/projected/f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be-kube-api-access-fs2ls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:13.510043 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:13.508516    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d" event={"ID":"264eaf78-ec83-4872-8a0b-e1de8f8e5a0b","Type":"ContainerDied","Data":"7cbd2a86c780ab6df20985994fb3132c9fd0f4472d758f8a56caa9548fdf96d9"}
Nov 03 20:19:13.510043 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:13.508631    4421 scope.go:122] "RemoveContainer" containerID="9c2b57a363542bc4542e31143ff641adbc3ee536f68b7fef147ed3667c7ece35"
Nov 03 20:19:13.510043 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:13.508826    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d"
Nov 03 20:19:13.525839 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:13.525809    4421 generic.go:356] "Generic (PLEG): container finished" podID="f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be" containerID="21ac8bc4582349cd23f23d4956eda0a0a1279dc23cc879b0f744ae04cb20999a" exitCode=0
Nov 03 20:19:13.525978 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:13.525965    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l" event={"ID":"f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be","Type":"ContainerDied","Data":"21ac8bc4582349cd23f23d4956eda0a0a1279dc23cc879b0f744ae04cb20999a"}
Nov 03 20:19:13.526020 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:13.526003    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l" event={"ID":"f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be","Type":"ContainerDied","Data":"42743822f858a62374b18a5cee442e28c53c56301d25c370c45b5808d0c83fb7"}
Nov 03 20:19:13.526064 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:13.526057    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l"
Nov 03 20:19:13.539063 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:13.539043    4421 scope.go:122] "RemoveContainer" containerID="21ac8bc4582349cd23f23d4956eda0a0a1279dc23cc879b0f744ae04cb20999a"
Nov 03 20:19:13.549416 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:13.549387    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d"]
Nov 03 20:19:13.552143 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:13.552122    4421 scope.go:122] "RemoveContainer" containerID="21ac8bc4582349cd23f23d4956eda0a0a1279dc23cc879b0f744ae04cb20999a"
Nov 03 20:19:13.552481 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:19:13.552459    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21ac8bc4582349cd23f23d4956eda0a0a1279dc23cc879b0f744ae04cb20999a\": container with ID starting with 21ac8bc4582349cd23f23d4956eda0a0a1279dc23cc879b0f744ae04cb20999a not found: ID does not exist" containerID="21ac8bc4582349cd23f23d4956eda0a0a1279dc23cc879b0f744ae04cb20999a"
Nov 03 20:19:13.552550 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:13.552489    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21ac8bc4582349cd23f23d4956eda0a0a1279dc23cc879b0f744ae04cb20999a"} err="failed to get container status \"21ac8bc4582349cd23f23d4956eda0a0a1279dc23cc879b0f744ae04cb20999a\": rpc error: code = NotFound desc = could not find container \"21ac8bc4582349cd23f23d4956eda0a0a1279dc23cc879b0f744ae04cb20999a\": container with ID starting with 21ac8bc4582349cd23f23d4956eda0a0a1279dc23cc879b0f744ae04cb20999a not found: ID does not exist"
Nov 03 20:19:13.553047 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:13.553027    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-98148a-6576c6f578-9kc9d"]
Nov 03 20:19:13.567966 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:13.567945    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l"]
Nov 03 20:19:13.571855 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:13.571831    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-98148a-7f86c6bb8d-hnn6l"]
Nov 03 20:19:14.031547 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:14.031506    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="264eaf78-ec83-4872-8a0b-e1de8f8e5a0b" path="/var/lib/kubelet/pods/264eaf78-ec83-4872-8a0b-e1de8f8e5a0b/volumes"
Nov 03 20:19:14.054189 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:14.032237    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be" path="/var/lib/kubelet/pods/f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be/volumes"
Nov 03 20:19:16.253917 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.245359    4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:16.444623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.434814    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"]
Nov 03 20:19:16.444623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.435802    4421 cpu_manager.go:414] "containerMap: removing container" podUID="6feb18b2-b54f-4850-b7d0-59c872386051" containerName="istio-proxy"
Nov 03 20:19:16.444623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.435824    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6feb18b2-b54f-4850-b7d0-59c872386051" containerName="istio-proxy"
Nov 03 20:19:16.444623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.435866    4421 cpu_manager.go:414] "containerMap: removing container" podUID="264eaf78-ec83-4872-8a0b-e1de8f8e5a0b" containerName="maas-api"
Nov 03 20:19:16.444623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.435875    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="264eaf78-ec83-4872-8a0b-e1de8f8e5a0b" containerName="maas-api"
Nov 03 20:19:16.444623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.435898    4421 cpu_manager.go:414] "containerMap: removing container" podUID="f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be" containerName="payload-processing"
Nov 03 20:19:16.444623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.435906    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be" containerName="payload-processing"
Nov 03 20:19:16.444623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.435921    4421 cpu_manager.go:414] "containerMap: removing container" podUID="acbca941-61d7-4a7a-86d9-76494049a25d" containerName="istio-proxy"
Nov 03 20:19:16.444623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.435930    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acbca941-61d7-4a7a-86d9-76494049a25d" containerName="istio-proxy"
Nov 03 20:19:16.444623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.436078    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="acbca941-61d7-4a7a-86d9-76494049a25d" containerName="istio-proxy"
Nov 03 20:19:16.444623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.436094    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="264eaf78-ec83-4872-8a0b-e1de8f8e5a0b" containerName="maas-api"
Nov 03 20:19:16.444623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.436107    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f583fa6c-ff3a-4af6-8b3f-2e3de29fa4be" containerName="payload-processing"
Nov 03 20:19:16.444623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.436118    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6feb18b2-b54f-4850-b7d0-59c872386051" containerName="istio-proxy"
Nov 03 20:19:16.474622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.464443    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.474622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.464497    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"]
Nov 03 20:19:16.581785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.541576    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5488adb-f734-4d16-81bf-f230fca1306f-istio-podinfo\") pod \"e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") " pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.581785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.563540    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-istio-data\") pod \"e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") " pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.581785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.563680    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-workload-socket\") pod \"e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") " pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.581785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.563723    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-istio-envoy\") pod \"e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") " pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.581785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.563753    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzq7x\" (UniqueName: \"kubernetes.io/projected/b5488adb-f734-4d16-81bf-f230fca1306f-kube-api-access-dzq7x\") pod \"e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") " pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.581785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.563868    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5488adb-f734-4d16-81bf-f230fca1306f-istio-token\") pod \"e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") " pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.581785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.563934    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-workload-certs\") pod \"e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") " pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.581785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.564060    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5488adb-f734-4d16-81bf-f230fca1306f-istiod-ca-cert\") pod \"e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") " pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.581785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.564213    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-credential-socket\") pod \"e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") " pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.581785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.543937    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9a9ea9-openshift-default-dockercfg-4tnw4\""
Nov 03 20:19:16.581785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.566964    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:16.627319 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.627234    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:16.673622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.665495    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5488adb-f734-4d16-81bf-f230fca1306f-istiod-ca-cert\") pod \"e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") " pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.700634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.697244    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-credential-socket\") pod \"e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") " pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.700634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.697357    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5488adb-f734-4d16-81bf-f230fca1306f-istio-podinfo\") pod \"e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") " pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.700634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.697404    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-istio-data\") pod \"e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") " pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.700634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.697506    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-workload-socket\") pod \"e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") " pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.700634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.697547    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-istio-envoy\") pod \"e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") " pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.700634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.697576    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzq7x\" (UniqueName: \"kubernetes.io/projected/b5488adb-f734-4d16-81bf-f230fca1306f-kube-api-access-dzq7x\") pod \"e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") " pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.700634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.697682    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5488adb-f734-4d16-81bf-f230fca1306f-istio-token\") pod \"e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") " pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.700634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.697747    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-workload-certs\") pod \"e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") " pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.700634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.698300    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-workload-certs\") pod \"e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") " pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.700634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.687774    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5488adb-f734-4d16-81bf-f230fca1306f-istiod-ca-cert\") pod \"e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") " pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.700634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.698576    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-credential-socket\") pod \"e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") " pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.718120 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.717275    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-workload-socket\") pod \"e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") " pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.719339 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.719281    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-istio-data\") pod \"e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") " pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.728564 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.728513    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzq7x\" (UniqueName: \"kubernetes.io/projected/b5488adb-f734-4d16-81bf-f230fca1306f-kube-api-access-dzq7x\") pod \"e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") " pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.742284 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.742254    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5488adb-f734-4d16-81bf-f230fca1306f-istio-podinfo\") pod \"e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") " pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.743158 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.743129    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5488adb-f734-4d16-81bf-f230fca1306f-istio-token\") pod \"e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") " pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.743575 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.743553    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-istio-envoy\") pod \"e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") " pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.776716 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.775371    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:16.893264 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.892901    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:16.970612 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:16.969458    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:17.158032 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:17.156799    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:17.295623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:17.295264    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:18.115625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:18.062858    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:18.162178 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:18.161008    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:18.338642 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:18.336900    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"]
Nov 03 20:19:18.338642 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:18.336963    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:19:18.338642 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:18.336990    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:19:18.338642 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:18.337013    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:19:18.350189 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:18.350016    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:19:18.428456 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:18.423437    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:19:18.428456 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:18.423523    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:19:18.428456 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:18.423563    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:19:19.131835 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:19.131792    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln" event={"ID":"b5488adb-f734-4d16-81bf-f230fca1306f","Type":"ContainerStarted","Data":"bd20d3285ea2dbd64059ed5b469e184320bfb51da263079d47d02b26fd6b69bd"}
Nov 03 20:19:19.132037 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:19.131875    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln" event={"ID":"b5488adb-f734-4d16-81bf-f230fca1306f","Type":"ContainerStarted","Data":"2d07ac361ba739400938b903b3e29aaf1aba98b8353e0d00a89fa0f96ac5e082"}
Nov 03 20:19:19.259624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:19.229251    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln" podStartSLOduration=4.229230906 podStartE2EDuration="4.229230906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:19.209956592 +0000 UTC m=+1089.343085054" watchObservedRunningTime="2026-11-03 20:19:19.229230906 +0000 UTC m=+1089.362359369"
Nov 03 20:19:19.722796 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:19:19.722759    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:19:19.894981 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:19.894953    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:19.901132 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:19.901087    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:20.200285 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:20.200251    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:20.208994 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:20.208961    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:22.602406 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:22.602375    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk"]
Nov 03 20:19:22.606756 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:22.606735    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk"
Nov 03 20:19:22.612882 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:22.612856    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-9a9ea9\""
Nov 03 20:19:22.613028 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:22.612857    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-9a9ea9-dockercfg-qpq9g\""
Nov 03 20:19:22.625263 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:22.625236    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk"]
Nov 03 20:19:22.630896 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:22.630869    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5"]
Nov 03 20:19:22.638305 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:22.638278    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5"
Nov 03 20:19:22.650464 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:22.650438    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5"]
Nov 03 20:19:22.681970 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:22.681918    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgrq9\" (UniqueName: \"kubernetes.io/projected/a1903a77-2443-4c41-984c-45b71e22201a-kube-api-access-kgrq9\") pod \"payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk\" (UID: \"a1903a77-2443-4c41-984c-45b71e22201a\") " pod="openshift-ingress/payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk"
Nov 03 20:19:22.682659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:22.682613    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1903a77-2443-4c41-984c-45b71e22201a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk\" (UID: \"a1903a77-2443-4c41-984c-45b71e22201a\") " pod="openshift-ingress/payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk"
Nov 03 20:19:22.879227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:22.796913    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a087d02d-46f7-4eb1-b085-52f542894f70-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5\" (UID: \"a087d02d-46f7-4eb1-b085-52f542894f70\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5"
Nov 03 20:19:22.879227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:22.797062    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgrq9\" (UniqueName: \"kubernetes.io/projected/a1903a77-2443-4c41-984c-45b71e22201a-kube-api-access-kgrq9\") pod \"payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk\" (UID: \"a1903a77-2443-4c41-984c-45b71e22201a\") " pod="openshift-ingress/payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk"
Nov 03 20:19:22.879227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:22.797102    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmdvn\" (UniqueName: \"kubernetes.io/projected/a087d02d-46f7-4eb1-b085-52f542894f70-kube-api-access-zmdvn\") pod \"payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5\" (UID: \"a087d02d-46f7-4eb1-b085-52f542894f70\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5"
Nov 03 20:19:22.879227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:22.797253    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1903a77-2443-4c41-984c-45b71e22201a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk\" (UID: \"a1903a77-2443-4c41-984c-45b71e22201a\") " pod="openshift-ingress/payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk"
Nov 03 20:19:22.879227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:22.798295    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1903a77-2443-4c41-984c-45b71e22201a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk\" (UID: \"a1903a77-2443-4c41-984c-45b71e22201a\") " pod="openshift-ingress/payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk"
Nov 03 20:19:22.901677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:22.901634    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmdvn\" (UniqueName: \"kubernetes.io/projected/a087d02d-46f7-4eb1-b085-52f542894f70-kube-api-access-zmdvn\") pod \"payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5\" (UID: \"a087d02d-46f7-4eb1-b085-52f542894f70\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5"
Nov 03 20:19:22.901986 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:22.901959    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a087d02d-46f7-4eb1-b085-52f542894f70-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5\" (UID: \"a087d02d-46f7-4eb1-b085-52f542894f70\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5"
Nov 03 20:19:22.912080 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:22.902880    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a087d02d-46f7-4eb1-b085-52f542894f70-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5\" (UID: \"a087d02d-46f7-4eb1-b085-52f542894f70\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5"
Nov 03 20:19:22.926932 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:22.923271    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgrq9\" (UniqueName: \"kubernetes.io/projected/a1903a77-2443-4c41-984c-45b71e22201a-kube-api-access-kgrq9\") pod \"payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk\" (UID: \"a1903a77-2443-4c41-984c-45b71e22201a\") " pod="openshift-ingress/payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk"
Nov 03 20:19:22.926932 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:22.926896    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk"
Nov 03 20:19:22.955379 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:22.955339    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmdvn\" (UniqueName: \"kubernetes.io/projected/a087d02d-46f7-4eb1-b085-52f542894f70-kube-api-access-zmdvn\") pod \"payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5\" (UID: \"a087d02d-46f7-4eb1-b085-52f542894f70\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5"
Nov 03 20:19:22.998354 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:22.998139    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5"
Nov 03 20:19:23.341902 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:23.341563    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5"]
Nov 03 20:19:23.347935 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:23.347590    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk"]
Nov 03 20:19:23.568289 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:23.568252    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5"]
Nov 03 20:19:23.581812 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:23.581767    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz"]
Nov 03 20:19:24.245390 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:24.245355    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk" event={"ID":"a1903a77-2443-4c41-984c-45b71e22201a","Type":"ContainerStarted","Data":"a687832d0a0144ee51c273d964e2318447f620f1a601800c74fbd98171abcd65"}
Nov 03 20:19:24.245907 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:24.245416    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk" event={"ID":"a1903a77-2443-4c41-984c-45b71e22201a","Type":"ContainerStarted","Data":"5f855635fc08d1f3480b77850af5b001d9fbc893030ca0bc9c8d92c31b025997"}
Nov 03 20:19:24.245907 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:24.245468    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk"
Nov 03 20:19:24.247456 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:24.247433    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5" event={"ID":"a087d02d-46f7-4eb1-b085-52f542894f70","Type":"ContainerStarted","Data":"77f6c28ffcc2e0e7e35957097d14b2488f453351c83127789d6c30bdcc3e5128"}
Nov 03 20:19:24.247582 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:24.247468    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5" event={"ID":"a087d02d-46f7-4eb1-b085-52f542894f70","Type":"ContainerStarted","Data":"65f10ca2fcd1096999458c2ed4cb3de8239d899b5dd9b8c0c83a0d95299053fc"}
Nov 03 20:19:24.247582 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:24.247571    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5"
Nov 03 20:19:24.267450 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:24.267381    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk" podStartSLOduration=2.267363555 podStartE2EDuration="2.267363555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:24.263415087 +0000 UTC m=+1094.396543553" watchObservedRunningTime="2026-11-03 20:19:24.267363555 +0000 UTC m=+1094.400492018"
Nov 03 20:19:24.289403 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:24.287958    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5" podStartSLOduration=2.287941895 podStartE2EDuration="2.287941895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:24.284144482 +0000 UTC m=+1094.417272945" watchObservedRunningTime="2026-11-03 20:19:24.287941895 +0000 UTC m=+1094.421070387"
Nov 03 20:19:26.265179 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:26.265120    4421 generic.go:356] "Generic (PLEG): container finished" podID="1a045c5d-d469-415e-8fc1-921f2ff05253" containerID="61dfbbabb674ae282203f19adf30517d5d084158901086054ca0032998cd158f" exitCode=0
Nov 03 20:19:26.265179 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:26.265152    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9" event={"ID":"1a045c5d-d469-415e-8fc1-921f2ff05253","Type":"ContainerDied","Data":"61dfbbabb674ae282203f19adf30517d5d084158901086054ca0032998cd158f"}
Nov 03 20:19:26.265719 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:26.265702    4421 scope.go:122] "RemoveContainer" containerID="61dfbbabb674ae282203f19adf30517d5d084158901086054ca0032998cd158f"
Nov 03 20:19:26.267540 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:26.267510    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:19:26.267659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:26.267578    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:19:26.267659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:26.267644    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:19:27.278634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:27.278565    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9" event={"ID":"1a045c5d-d469-415e-8fc1-921f2ff05253","Type":"ContainerStarted","Data":"3172749164bac549f74cf9aab96ff3daafd413e737da4bc0f33c79c6dfb7be63"}
Nov 03 20:19:27.729996 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:27.729957    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:19:27.730213 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:27.730133    4421 patch_prober.go:29] interesting pod/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 03 20:19:27.730213 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:27.730190    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9" podUID="1a045c5d-d469-415e-8fc1-921f2ff05253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 03 20:19:27.875738 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:27.875700    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn"]
Nov 03 20:19:27.887648 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:27.885851    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn"]
Nov 03 20:19:27.887648 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:27.885973    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn"
Nov 03 20:19:27.890790 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:27.890726    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-9a9ea9-serving-cert\""
Nov 03 20:19:27.917172 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:27.917139    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9e5419-faf0-42aa-befb-79149b3cd0f1-maas-api-tls\") pod \"maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn\" (UID: \"6c9e5419-faf0-42aa-befb-79149b3cd0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn"
Nov 03 20:19:27.917315 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:27.917198    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4lfs\" (UniqueName: \"kubernetes.io/projected/6c9e5419-faf0-42aa-befb-79149b3cd0f1-kube-api-access-x4lfs\") pod \"maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn\" (UID: \"6c9e5419-faf0-42aa-befb-79149b3cd0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn"
Nov 03 20:19:27.917527 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:27.917501    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c9e5419-faf0-42aa-befb-79149b3cd0f1-openshift-service-ca\") pod \"maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn\" (UID: \"6c9e5419-faf0-42aa-befb-79149b3cd0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn"
Nov 03 20:19:27.917688 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:27.917668    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9e5419-faf0-42aa-befb-79149b3cd0f1-metrics-certs\") pod \"maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn\" (UID: \"6c9e5419-faf0-42aa-befb-79149b3cd0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn"
Nov 03 20:19:28.052680 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:28.034021    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c9e5419-faf0-42aa-befb-79149b3cd0f1-openshift-service-ca\") pod \"maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn\" (UID: \"6c9e5419-faf0-42aa-befb-79149b3cd0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn"
Nov 03 20:19:28.070001 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:28.069967    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9e5419-faf0-42aa-befb-79149b3cd0f1-metrics-certs\") pod \"maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn\" (UID: \"6c9e5419-faf0-42aa-befb-79149b3cd0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn"
Nov 03 20:19:28.084897 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:28.084861    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9e5419-faf0-42aa-befb-79149b3cd0f1-maas-api-tls\") pod \"maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn\" (UID: \"6c9e5419-faf0-42aa-befb-79149b3cd0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn"
Nov 03 20:19:28.085184 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:28.085160    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4lfs\" (UniqueName: \"kubernetes.io/projected/6c9e5419-faf0-42aa-befb-79149b3cd0f1-kube-api-access-x4lfs\") pod \"maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn\" (UID: \"6c9e5419-faf0-42aa-befb-79149b3cd0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn"
Nov 03 20:19:28.100044 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:28.036156    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c9e5419-faf0-42aa-befb-79149b3cd0f1-openshift-service-ca\") pod \"maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn\" (UID: \"6c9e5419-faf0-42aa-befb-79149b3cd0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn"
Nov 03 20:19:28.122628 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:19:28.116957    4421 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-serving-cert: secret "maas-api-e2e-api-a-9a9ea9-serving-cert" not found
Nov 03 20:19:28.122628 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:19:28.117038    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c9e5419-faf0-42aa-befb-79149b3cd0f1-maas-api-tls podName:6c9e5419-faf0-42aa-befb-79149b3cd0f1 nodeName:}" failed. No retries permitted until 2026-11-03 20:19:28.617017731 +0000 UTC m=+1098.750146173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6c9e5419-faf0-42aa-befb-79149b3cd0f1-maas-api-tls") pod "maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn" (UID: "6c9e5419-faf0-42aa-befb-79149b3cd0f1") : secret "maas-api-e2e-api-a-9a9ea9-serving-cert" not found
Nov 03 20:19:28.128308 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:28.128137    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9e5419-faf0-42aa-befb-79149b3cd0f1-metrics-certs\") pod \"maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn\" (UID: \"6c9e5419-faf0-42aa-befb-79149b3cd0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn"
Nov 03 20:19:28.130136 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:28.129342    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4lfs\" (UniqueName: \"kubernetes.io/projected/6c9e5419-faf0-42aa-befb-79149b3cd0f1-kube-api-access-x4lfs\") pod \"maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn\" (UID: \"6c9e5419-faf0-42aa-befb-79149b3cd0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn"
Nov 03 20:19:28.581621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:28.577682    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5" podUID="cac44a07-d8cf-47e4-93d0-b237eb36b446" containerName="payload-processing" containerID="cri-o://c27864dfd7c435e219a18a253074264e84776e29798e0ea43274121935d4af9a" gracePeriod=25
Nov 03 20:19:28.591088 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:28.589019    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz" podUID="55a43d29-6ffd-484b-8dcc-a827785a46f1" containerName="payload-pre-processing" containerID="cri-o://919e442fcd12eacc4118fdb9afabd8b67cabd5f2b24c49fd309eb3d81ad7c8b5" gracePeriod=25
Nov 03 20:19:28.707665 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:28.696714    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9e5419-faf0-42aa-befb-79149b3cd0f1-maas-api-tls\") pod \"maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn\" (UID: \"6c9e5419-faf0-42aa-befb-79149b3cd0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn"
Nov 03 20:19:28.744931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:28.740654    4421 patch_prober.go:29] interesting pod/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 03 20:19:28.744931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:28.740710    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9" podUID="1a045c5d-d469-415e-8fc1-921f2ff05253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 03 20:19:28.744931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:28.742215    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck"]
Nov 03 20:19:28.744931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:28.742473    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck" podUID="aad3c734-b6ba-42f2-a3f2-6794d4e86793" containerName="maas-api" containerID="cri-o://a26cd0f3349a7315f762d4a7933677b89ab2bbf45bf205fd958de6151ed157cf" gracePeriod=30
Nov 03 20:19:28.771186 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:28.771154    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9e5419-faf0-42aa-befb-79149b3cd0f1-maas-api-tls\") pod \"maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn\" (UID: \"6c9e5419-faf0-42aa-befb-79149b3cd0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn"
Nov 03 20:19:28.817071 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:28.817047    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn"
Nov 03 20:19:29.332745 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.332701    4421 generic.go:356] "Generic (PLEG): container finished" podID="aad3c734-b6ba-42f2-a3f2-6794d4e86793" containerID="a26cd0f3349a7315f762d4a7933677b89ab2bbf45bf205fd958de6151ed157cf" exitCode=0
Nov 03 20:19:29.332972 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.332938    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck" event={"ID":"aad3c734-b6ba-42f2-a3f2-6794d4e86793","Type":"ContainerDied","Data":"a26cd0f3349a7315f762d4a7933677b89ab2bbf45bf205fd958de6151ed157cf"}
Nov 03 20:19:29.336246 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.336220    4421 generic.go:356] "Generic (PLEG): container finished" podID="55a43d29-6ffd-484b-8dcc-a827785a46f1" containerID="919e442fcd12eacc4118fdb9afabd8b67cabd5f2b24c49fd309eb3d81ad7c8b5" exitCode=0
Nov 03 20:19:29.338782 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.338761    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz" event={"ID":"55a43d29-6ffd-484b-8dcc-a827785a46f1","Type":"ContainerDied","Data":"919e442fcd12eacc4118fdb9afabd8b67cabd5f2b24c49fd309eb3d81ad7c8b5"}
Nov 03 20:19:29.397063 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.396913    4421 generic.go:356] "Generic (PLEG): container finished" podID="cac44a07-d8cf-47e4-93d0-b237eb36b446" containerID="c27864dfd7c435e219a18a253074264e84776e29798e0ea43274121935d4af9a" exitCode=0
Nov 03 20:19:29.397063 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.397015    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5" event={"ID":"cac44a07-d8cf-47e4-93d0-b237eb36b446","Type":"ContainerDied","Data":"c27864dfd7c435e219a18a253074264e84776e29798e0ea43274121935d4af9a"}
Nov 03 20:19:29.406520 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.405802    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz"
Nov 03 20:19:29.507620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.496465    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55a43d29-6ffd-484b-8dcc-a827785a46f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55a43d29-6ffd-484b-8dcc-a827785a46f1-plugins-config-volume\") pod \"55a43d29-6ffd-484b-8dcc-a827785a46f1\" (UID: \"55a43d29-6ffd-484b-8dcc-a827785a46f1\") "
Nov 03 20:19:29.507620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.496727    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55a43d29-6ffd-484b-8dcc-a827785a46f1-kube-api-access-t6wdn\" (UniqueName: \"kubernetes.io/projected/55a43d29-6ffd-484b-8dcc-a827785a46f1-kube-api-access-t6wdn\") pod \"55a43d29-6ffd-484b-8dcc-a827785a46f1\" (UID: \"55a43d29-6ffd-484b-8dcc-a827785a46f1\") "
Nov 03 20:19:29.507620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.498125    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55a43d29-6ffd-484b-8dcc-a827785a46f1-plugins-config-volume" pod "55a43d29-6ffd-484b-8dcc-a827785a46f1" (UID: "55a43d29-6ffd-484b-8dcc-a827785a46f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:29.507620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.498655    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn"]
Nov 03 20:19:29.520798 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.520764    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a43d29-6ffd-484b-8dcc-a827785a46f1-kube-api-access-t6wdn" pod "55a43d29-6ffd-484b-8dcc-a827785a46f1" (UID: "55a43d29-6ffd-484b-8dcc-a827785a46f1"). InnerVolumeSpecName "kube-api-access-t6wdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:29.601684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.601590    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55a43d29-6ffd-484b-8dcc-a827785a46f1-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.601684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.601651    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6wdn\" (UniqueName: \"kubernetes.io/projected/55a43d29-6ffd-484b-8dcc-a827785a46f1-kube-api-access-t6wdn\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.610154 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.610130    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5"
Nov 03 20:19:29.625210 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.625186    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck"
Nov 03 20:19:29.702694 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.702665    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aad3c734-b6ba-42f2-a3f2-6794d4e86793-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aad3c734-b6ba-42f2-a3f2-6794d4e86793-metrics-certs\") pod \"aad3c734-b6ba-42f2-a3f2-6794d4e86793\" (UID: \"aad3c734-b6ba-42f2-a3f2-6794d4e86793\") "
Nov 03 20:19:29.702888 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.702760    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cac44a07-d8cf-47e4-93d0-b237eb36b446-kube-api-access-jp2pf\" (UniqueName: \"kubernetes.io/projected/cac44a07-d8cf-47e4-93d0-b237eb36b446-kube-api-access-jp2pf\") pod \"cac44a07-d8cf-47e4-93d0-b237eb36b446\" (UID: \"cac44a07-d8cf-47e4-93d0-b237eb36b446\") "
Nov 03 20:19:29.702888 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.702815    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aad3c734-b6ba-42f2-a3f2-6794d4e86793-kube-api-access-lt9pm\" (UniqueName: \"kubernetes.io/projected/aad3c734-b6ba-42f2-a3f2-6794d4e86793-kube-api-access-lt9pm\") pod \"aad3c734-b6ba-42f2-a3f2-6794d4e86793\" (UID: \"aad3c734-b6ba-42f2-a3f2-6794d4e86793\") "
Nov 03 20:19:29.703072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.702949    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cac44a07-d8cf-47e4-93d0-b237eb36b446-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cac44a07-d8cf-47e4-93d0-b237eb36b446-plugins-config-volume\") pod \"cac44a07-d8cf-47e4-93d0-b237eb36b446\" (UID: \"cac44a07-d8cf-47e4-93d0-b237eb36b446\") "
Nov 03 20:19:29.703072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.702982    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aad3c734-b6ba-42f2-a3f2-6794d4e86793-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aad3c734-b6ba-42f2-a3f2-6794d4e86793-openshift-service-ca\") pod \"aad3c734-b6ba-42f2-a3f2-6794d4e86793\" (UID: \"aad3c734-b6ba-42f2-a3f2-6794d4e86793\") "
Nov 03 20:19:29.703179 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.703129    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aad3c734-b6ba-42f2-a3f2-6794d4e86793-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aad3c734-b6ba-42f2-a3f2-6794d4e86793-maas-api-tls\") pod \"aad3c734-b6ba-42f2-a3f2-6794d4e86793\" (UID: \"aad3c734-b6ba-42f2-a3f2-6794d4e86793\") "
Nov 03 20:19:29.703780 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.703298    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad3c734-b6ba-42f2-a3f2-6794d4e86793-openshift-service-ca" pod "aad3c734-b6ba-42f2-a3f2-6794d4e86793" (UID: "aad3c734-b6ba-42f2-a3f2-6794d4e86793"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:29.703780 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.703358    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cac44a07-d8cf-47e4-93d0-b237eb36b446-plugins-config-volume" pod "cac44a07-d8cf-47e4-93d0-b237eb36b446" (UID: "cac44a07-d8cf-47e4-93d0-b237eb36b446"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:29.704131 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.704086    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cac44a07-d8cf-47e4-93d0-b237eb36b446-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.704131 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.704116    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aad3c734-b6ba-42f2-a3f2-6794d4e86793-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.705312 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.705292    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad3c734-b6ba-42f2-a3f2-6794d4e86793-kube-api-access-lt9pm" pod "aad3c734-b6ba-42f2-a3f2-6794d4e86793" (UID: "aad3c734-b6ba-42f2-a3f2-6794d4e86793"). InnerVolumeSpecName "kube-api-access-lt9pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:29.705404 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.705317    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aad3c734-b6ba-42f2-a3f2-6794d4e86793-metrics-certs" pod "aad3c734-b6ba-42f2-a3f2-6794d4e86793" (UID: "aad3c734-b6ba-42f2-a3f2-6794d4e86793"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:19:29.705404 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.705325    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac44a07-d8cf-47e4-93d0-b237eb36b446-kube-api-access-jp2pf" pod "cac44a07-d8cf-47e4-93d0-b237eb36b446" (UID: "cac44a07-d8cf-47e4-93d0-b237eb36b446"). InnerVolumeSpecName "kube-api-access-jp2pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:29.705729 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.705714    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aad3c734-b6ba-42f2-a3f2-6794d4e86793-maas-api-tls" pod "aad3c734-b6ba-42f2-a3f2-6794d4e86793" (UID: "aad3c734-b6ba-42f2-a3f2-6794d4e86793"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:19:29.731239 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.731210    4421 patch_prober.go:29] interesting pod/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 03 20:19:29.731358 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.731261    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9" podUID="1a045c5d-d469-415e-8fc1-921f2ff05253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 03 20:19:29.805496 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.805467    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aad3c734-b6ba-42f2-a3f2-6794d4e86793-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.805496 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.805494    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aad3c734-b6ba-42f2-a3f2-6794d4e86793-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.805496 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.805504    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp2pf\" (UniqueName: \"kubernetes.io/projected/cac44a07-d8cf-47e4-93d0-b237eb36b446-kube-api-access-jp2pf\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:29.805751 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:29.805513    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt9pm\" (UniqueName: \"kubernetes.io/projected/aad3c734-b6ba-42f2-a3f2-6794d4e86793-kube-api-access-lt9pm\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:30.448627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:30.444948    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck" event={"ID":"aad3c734-b6ba-42f2-a3f2-6794d4e86793","Type":"ContainerDied","Data":"264b6f444a93faeb3664ca77b866e3e16fb4a293c0dba762e4f7a2c2b40a1eed"}
Nov 03 20:19:30.448627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:30.445030    4421 scope.go:122] "RemoveContainer" containerID="a26cd0f3349a7315f762d4a7933677b89ab2bbf45bf205fd958de6151ed157cf"
Nov 03 20:19:30.448627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:30.445228    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck"
Nov 03 20:19:30.474993 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:30.474962    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn" event={"ID":"6c9e5419-faf0-42aa-befb-79149b3cd0f1","Type":"ContainerStarted","Data":"94ba640b5d22ebb632c7aa9f78d29859ba6759126c2f7569d33f6d21c67aa315"}
Nov 03 20:19:30.475290 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:30.475207    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn" event={"ID":"6c9e5419-faf0-42aa-befb-79149b3cd0f1","Type":"ContainerStarted","Data":"7fd148ea06b9a3ec96c08f11d1458e7e4ab1e671a000b8b6ad6fd66d30ffac02"}
Nov 03 20:19:30.475408 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:30.475398    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn"
Nov 03 20:19:30.476808 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:30.476785    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz" event={"ID":"55a43d29-6ffd-484b-8dcc-a827785a46f1","Type":"ContainerDied","Data":"526d7ea22f7d37c3183a66f15d46df8cc468f2ca03bfafef6af8ae23af651723"}
Nov 03 20:19:30.476961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:30.476946    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz"
Nov 03 20:19:30.478917 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:30.478896    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5" event={"ID":"cac44a07-d8cf-47e4-93d0-b237eb36b446","Type":"ContainerDied","Data":"a747d1d269fa9e45b63fa38a3071746c8eb7b670c255f532f9b020af3106477b"}
Nov 03 20:19:30.478917 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:30.478902    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5"
Nov 03 20:19:30.483132 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:30.483117    4421 scope.go:122] "RemoveContainer" containerID="919e442fcd12eacc4118fdb9afabd8b67cabd5f2b24c49fd309eb3d81ad7c8b5"
Nov 03 20:19:30.492730 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:30.492707    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck"]
Nov 03 20:19:30.495521 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:30.495497    4421 scope.go:122] "RemoveContainer" containerID="c27864dfd7c435e219a18a253074264e84776e29798e0ea43274121935d4af9a"
Nov 03 20:19:30.497937 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:30.497919    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-33652a-976894fcb-j9pck"]
Nov 03 20:19:30.513882 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:30.513839    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn" podStartSLOduration=2.956023517 podStartE2EDuration="3.513824752s" totalImagesPullingTime="557.801235ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:30.511069015 +0000 UTC m=+1100.644197478" watchObservedRunningTime="2026-11-03 20:19:30.513824752 +0000 UTC m=+1100.646953217"
Nov 03 20:19:30.527756 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:30.527733    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5"]
Nov 03 20:19:30.532018 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:30.531999    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-33652a-854c8bd4b8-g2cb5"]
Nov 03 20:19:30.571233 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:30.571200    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz"]
Nov 03 20:19:30.574308 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:30.574287    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-33652a-7f5ffd9679-mhrbz"]
Nov 03 20:19:30.732366 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:30.732258    4421 patch_prober.go:29] interesting pod/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 03 20:19:30.732366 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:30.732325    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9" podUID="1a045c5d-d469-415e-8fc1-921f2ff05253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 03 20:19:31.729922 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:31.729892    4421 patch_prober.go:29] interesting pod/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 03 20:19:31.730106 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:31.729951    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9" podUID="1a045c5d-d469-415e-8fc1-921f2ff05253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 03 20:19:31.898282 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:31.898251    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"]
Nov 03 20:19:31.898680 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:31.898441    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9" podUID="1a045c5d-d469-415e-8fc1-921f2ff05253" containerName="istio-proxy" containerID="cri-o://3172749164bac549f74cf9aab96ff3daafd413e737da4bc0f33c79c6dfb7be63" gracePeriod=30
Nov 03 20:19:32.225359 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:32.222277    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55a43d29-6ffd-484b-8dcc-a827785a46f1" path="/var/lib/kubelet/pods/55a43d29-6ffd-484b-8dcc-a827785a46f1/volumes"
Nov 03 20:19:32.225359 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:32.224047    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aad3c734-b6ba-42f2-a3f2-6794d4e86793" path="/var/lib/kubelet/pods/aad3c734-b6ba-42f2-a3f2-6794d4e86793/volumes"
Nov 03 20:19:32.225359 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:32.224783    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cac44a07-d8cf-47e4-93d0-b237eb36b446" path="/var/lib/kubelet/pods/cac44a07-d8cf-47e4-93d0-b237eb36b446/volumes"
Nov 03 20:19:34.539010 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:34.538977    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:19:35.258553 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:35.258525    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk"
Nov 03 20:19:35.258773 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:35.258592    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5"
Nov 03 20:19:37.522486 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:37.522462    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn"
Nov 03 20:19:38.285551 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.285527    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:19:38.403347 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.397664    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-credential-socket\") pod \"1a045c5d-d469-415e-8fc1-921f2ff05253\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") "
Nov 03 20:19:38.403347 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.397724    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-data\") pod \"1a045c5d-d469-415e-8fc1-921f2ff05253\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") "
Nov 03 20:19:38.403347 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.397770    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-workload-socket\") pod \"1a045c5d-d469-415e-8fc1-921f2ff05253\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") "
Nov 03 20:19:38.403347 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.397847    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-token\" (UniqueName: \"kubernetes.io/projected/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-token\") pod \"1a045c5d-d469-415e-8fc1-921f2ff05253\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") "
Nov 03 20:19:38.403347 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.397936    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a045c5d-d469-415e-8fc1-921f2ff05253-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a045c5d-d469-415e-8fc1-921f2ff05253-istiod-ca-cert\") pod \"1a045c5d-d469-415e-8fc1-921f2ff05253\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") "
Nov 03 20:19:38.403347 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.397998    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a045c5d-d469-415e-8fc1-921f2ff05253-kube-api-access-sf5hh\" (UniqueName: \"kubernetes.io/projected/1a045c5d-d469-415e-8fc1-921f2ff05253-kube-api-access-sf5hh\") pod \"1a045c5d-d469-415e-8fc1-921f2ff05253\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") "
Nov 03 20:19:38.403347 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.398034    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-envoy\") pod \"1a045c5d-d469-415e-8fc1-921f2ff05253\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") "
Nov 03 20:19:38.403347 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.398068    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-podinfo\") pod \"1a045c5d-d469-415e-8fc1-921f2ff05253\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") "
Nov 03 20:19:38.403347 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.398220    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-workload-certs\") pod \"1a045c5d-d469-415e-8fc1-921f2ff05253\" (UID: \"1a045c5d-d469-415e-8fc1-921f2ff05253\") "
Nov 03 20:19:38.418103 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.416640    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-workload-certs" pod "1a045c5d-d469-415e-8fc1-921f2ff05253" (UID: "1a045c5d-d469-415e-8fc1-921f2ff05253"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:38.418103 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.416925    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-data" pod "1a045c5d-d469-415e-8fc1-921f2ff05253" (UID: "1a045c5d-d469-415e-8fc1-921f2ff05253"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:38.418103 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.417118    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-credential-socket" pod "1a045c5d-d469-415e-8fc1-921f2ff05253" (UID: "1a045c5d-d469-415e-8fc1-921f2ff05253"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:38.419746 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.419269    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-workload-socket" pod "1a045c5d-d469-415e-8fc1-921f2ff05253" (UID: "1a045c5d-d469-415e-8fc1-921f2ff05253"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:38.420084 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.420046    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a045c5d-d469-415e-8fc1-921f2ff05253-istiod-ca-cert" pod "1a045c5d-d469-415e-8fc1-921f2ff05253" (UID: "1a045c5d-d469-415e-8fc1-921f2ff05253"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:38.423851 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.422493    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-envoy" pod "1a045c5d-d469-415e-8fc1-921f2ff05253" (UID: "1a045c5d-d469-415e-8fc1-921f2ff05253"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:19:38.426804 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.423972    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a045c5d-d469-415e-8fc1-921f2ff05253-kube-api-access-sf5hh" pod "1a045c5d-d469-415e-8fc1-921f2ff05253" (UID: "1a045c5d-d469-415e-8fc1-921f2ff05253"). InnerVolumeSpecName "kube-api-access-sf5hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:38.426804 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.424710    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-podinfo" pod "1a045c5d-d469-415e-8fc1-921f2ff05253" (UID: "1a045c5d-d469-415e-8fc1-921f2ff05253"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:19:38.428525 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.428503    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-token" pod "1a045c5d-d469-415e-8fc1-921f2ff05253" (UID: "1a045c5d-d469-415e-8fc1-921f2ff05253"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:38.499886 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.499853    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf5hh\" (UniqueName: \"kubernetes.io/projected/1a045c5d-d469-415e-8fc1-921f2ff05253-kube-api-access-sf5hh\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:38.499886 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.499881    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:38.499886 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.499891    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:38.500094 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.499901    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:38.500094 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.499909    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:38.500094 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.499918    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:38.500094 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.499926    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a045c5d-d469-415e-8fc1-921f2ff05253-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:38.500094 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.499934    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a045c5d-d469-415e-8fc1-921f2ff05253-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:38.500094 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.499942    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a045c5d-d469-415e-8fc1-921f2ff05253-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:38.561362 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.561331    4421 generic.go:356] "Generic (PLEG): container finished" podID="1a045c5d-d469-415e-8fc1-921f2ff05253" containerID="3172749164bac549f74cf9aab96ff3daafd413e737da4bc0f33c79c6dfb7be63" exitCode=0
Nov 03 20:19:38.561768 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.561384    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"
Nov 03 20:19:38.561768 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.561395    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9" event={"ID":"1a045c5d-d469-415e-8fc1-921f2ff05253","Type":"ContainerDied","Data":"3172749164bac549f74cf9aab96ff3daafd413e737da4bc0f33c79c6dfb7be63"}
Nov 03 20:19:38.561768 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.561439    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9" event={"ID":"1a045c5d-d469-415e-8fc1-921f2ff05253","Type":"ContainerDied","Data":"e85c264b31059723fb36d56d7efcfd52770e1bbf4f8664511732e6b7cd6f9893"}
Nov 03 20:19:38.561768 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.561461    4421 scope.go:122] "RemoveContainer" containerID="3172749164bac549f74cf9aab96ff3daafd413e737da4bc0f33c79c6dfb7be63"
Nov 03 20:19:38.572468 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.572450    4421 scope.go:122] "RemoveContainer" containerID="61dfbbabb674ae282203f19adf30517d5d084158901086054ca0032998cd158f"
Nov 03 20:19:38.581766 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.581737    4421 scope.go:122] "RemoveContainer" containerID="3172749164bac549f74cf9aab96ff3daafd413e737da4bc0f33c79c6dfb7be63"
Nov 03 20:19:38.582175 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:19:38.582147    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3172749164bac549f74cf9aab96ff3daafd413e737da4bc0f33c79c6dfb7be63\": container with ID starting with 3172749164bac549f74cf9aab96ff3daafd413e737da4bc0f33c79c6dfb7be63 not found: ID does not exist" containerID="3172749164bac549f74cf9aab96ff3daafd413e737da4bc0f33c79c6dfb7be63"
Nov 03 20:19:38.582332 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.582307    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3172749164bac549f74cf9aab96ff3daafd413e737da4bc0f33c79c6dfb7be63"} err="failed to get container status \"3172749164bac549f74cf9aab96ff3daafd413e737da4bc0f33c79c6dfb7be63\": rpc error: code = NotFound desc = could not find container \"3172749164bac549f74cf9aab96ff3daafd413e737da4bc0f33c79c6dfb7be63\": container with ID starting with 3172749164bac549f74cf9aab96ff3daafd413e737da4bc0f33c79c6dfb7be63 not found: ID does not exist"
Nov 03 20:19:38.582449 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.582430    4421 scope.go:122] "RemoveContainer" containerID="61dfbbabb674ae282203f19adf30517d5d084158901086054ca0032998cd158f"
Nov 03 20:19:38.583131 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:19:38.583107    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61dfbbabb674ae282203f19adf30517d5d084158901086054ca0032998cd158f\": container with ID starting with 61dfbbabb674ae282203f19adf30517d5d084158901086054ca0032998cd158f not found: ID does not exist" containerID="61dfbbabb674ae282203f19adf30517d5d084158901086054ca0032998cd158f"
Nov 03 20:19:38.583213 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.583137    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61dfbbabb674ae282203f19adf30517d5d084158901086054ca0032998cd158f"} err="failed to get container status \"61dfbbabb674ae282203f19adf30517d5d084158901086054ca0032998cd158f\": rpc error: code = NotFound desc = could not find container \"61dfbbabb674ae282203f19adf30517d5d084158901086054ca0032998cd158f\": container with ID starting with 61dfbbabb674ae282203f19adf30517d5d084158901086054ca0032998cd158f not found: ID does not exist"
Nov 03 20:19:38.586471 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.586446    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"]
Nov 03 20:19:38.590158 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:38.590138    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-33652a-openshift-default-5ff86849f9-f8rn9"]
Nov 03 20:19:40.018030 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:40.017994    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a045c5d-d469-415e-8fc1-921f2ff05253" path="/var/lib/kubelet/pods/1a045c5d-d469-415e-8fc1-921f2ff05253/volumes"
Nov 03 20:19:42.811467 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:42.811435    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp"]
Nov 03 20:19:44.892559 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:44.892526    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"]
Nov 03 20:19:44.894345 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:44.894311    4421 cpu_manager.go:414] "containerMap: removing container" podUID="1a045c5d-d469-415e-8fc1-921f2ff05253" containerName="istio-proxy"
Nov 03 20:19:44.894345 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:44.894347    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a045c5d-d469-415e-8fc1-921f2ff05253" containerName="istio-proxy"
Nov 03 20:19:44.894493 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:44.894367    4421 cpu_manager.go:414] "containerMap: removing container" podUID="55a43d29-6ffd-484b-8dcc-a827785a46f1" containerName="payload-pre-processing"
Nov 03 20:19:44.894493 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:44.894375    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55a43d29-6ffd-484b-8dcc-a827785a46f1" containerName="payload-pre-processing"
Nov 03 20:19:44.894493 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:44.894391    4421 cpu_manager.go:414] "containerMap: removing container" podUID="cac44a07-d8cf-47e4-93d0-b237eb36b446" containerName="payload-processing"
Nov 03 20:19:44.894493 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:44.894399    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cac44a07-d8cf-47e4-93d0-b237eb36b446" containerName="payload-processing"
Nov 03 20:19:44.894493 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:44.894413    4421 cpu_manager.go:414] "containerMap: removing container" podUID="aad3c734-b6ba-42f2-a3f2-6794d4e86793" containerName="maas-api"
Nov 03 20:19:44.894493 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:44.894421    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad3c734-b6ba-42f2-a3f2-6794d4e86793" containerName="maas-api"
Nov 03 20:19:44.894696 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:44.894634    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="55a43d29-6ffd-484b-8dcc-a827785a46f1" containerName="payload-pre-processing"
Nov 03 20:19:44.894696 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:44.894650    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad3c734-b6ba-42f2-a3f2-6794d4e86793" containerName="maas-api"
Nov 03 20:19:44.894696 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:44.894665    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a045c5d-d469-415e-8fc1-921f2ff05253" containerName="istio-proxy"
Nov 03 20:19:44.894696 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:44.894675    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a045c5d-d469-415e-8fc1-921f2ff05253" containerName="istio-proxy"
Nov 03 20:19:44.894696 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:44.894687    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="cac44a07-d8cf-47e4-93d0-b237eb36b446" containerName="payload-processing"
Nov 03 20:19:44.900022 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:44.900004    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:44.914621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:44.909529    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-6b9849-openshift-default-dockercfg-n7jfh\""
Nov 03 20:19:44.916759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:44.916731    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"]
Nov 03 20:19:44.973720 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:44.973688    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-credential-socket\") pod \"e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") " pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:44.973916 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:44.973735    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-token\") pod \"e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") " pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:44.973916 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:44.973768    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrmw7\" (UniqueName: \"kubernetes.io/projected/249f4a98-95d2-4a3c-9cfc-a6f00e214815-kube-api-access-wrmw7\") pod \"e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") " pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:44.973916 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:44.973796    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-workload-certs\") pod \"e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") " pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:44.973916 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:44.973837    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-podinfo\") pod \"e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") " pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:44.974137 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:44.973978    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-data\") pod \"e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") " pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:44.974137 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:44.974054    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istiod-ca-cert\") pod \"e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") " pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:44.974137 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:44.974112    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-envoy\") pod \"e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") " pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:44.974281 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:44.974219    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-workload-socket\") pod \"e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") " pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:45.090623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:45.078529    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istiod-ca-cert\") pod \"e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") " pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:45.090623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:45.078634    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-envoy\") pod \"e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") " pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:45.090623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:45.078738    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-workload-socket\") pod \"e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") " pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:45.090623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:45.078805    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-credential-socket\") pod \"e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") " pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:45.090623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:45.078839    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-token\") pod \"e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") " pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:45.090623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:45.078864    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrmw7\" (UniqueName: \"kubernetes.io/projected/249f4a98-95d2-4a3c-9cfc-a6f00e214815-kube-api-access-wrmw7\") pod \"e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") " pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:45.090623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:45.078894    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-workload-certs\") pod \"e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") " pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:45.090623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:45.078942    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-podinfo\") pod \"e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") " pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:45.090623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:45.079094    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-data\") pod \"e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") " pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:45.090623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:45.079584    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-data\") pod \"e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") " pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:45.090623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:45.080231    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istiod-ca-cert\") pod \"e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") " pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:45.096620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:45.091708    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-workload-certs\") pod \"e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") " pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:45.096620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:45.092258    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-workload-socket\") pod \"e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") " pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:45.122622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:45.097361    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-credential-socket\") pod \"e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") " pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:45.145275 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:45.145159    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-token\") pod \"e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") " pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:45.197661 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:45.194441    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrmw7\" (UniqueName: \"kubernetes.io/projected/249f4a98-95d2-4a3c-9cfc-a6f00e214815-kube-api-access-wrmw7\") pod \"e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") " pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:45.249661 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:45.240871    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-podinfo\") pod \"e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") " pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:45.249661 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:45.241218    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-envoy\") pod \"e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") " pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:45.250244 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:45.250225    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:45.733713 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:45.733681    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"]
Nov 03 20:19:45.734497 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:19:45.734471    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod249f4a98_95d2_4a3c_9cfc_a6f00e214815.slice/crio-6e188a11bbe4ddb328fe30c9b33546014bd69eea5529f3293d7ce1c5ebf2d0f9 WatchSource:0}: Error finding container 6e188a11bbe4ddb328fe30c9b33546014bd69eea5529f3293d7ce1c5ebf2d0f9: Status 404 returned error can't find the container with id 6e188a11bbe4ddb328fe30c9b33546014bd69eea5529f3293d7ce1c5ebf2d0f9
Nov 03 20:19:45.737400 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:45.737367    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:19:45.737495 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:45.737440    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:19:45.737495 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:45.737476    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:19:46.738943 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:46.738915    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn" event={"ID":"249f4a98-95d2-4a3c-9cfc-a6f00e214815","Type":"ContainerStarted","Data":"54c278f026ce64e088b0760f089faf490b21f1ae15859d5a59884713aea1e772"}
Nov 03 20:19:46.739393 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:46.739375    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn" event={"ID":"249f4a98-95d2-4a3c-9cfc-a6f00e214815","Type":"ContainerStarted","Data":"6e188a11bbe4ddb328fe30c9b33546014bd69eea5529f3293d7ce1c5ebf2d0f9"}
Nov 03 20:19:46.780759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:46.780639    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn" podStartSLOduration=2.780620238 podStartE2EDuration="2.780620238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:46.774080263 +0000 UTC m=+1116.907208725" watchObservedRunningTime="2026-11-03 20:19:46.780620238 +0000 UTC m=+1116.913748701"
Nov 03 20:19:47.251024 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:47.250985    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:47.257116 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:47.257090    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:47.749225 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:47.749194    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:47.750158 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:47.750138    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:19:47.818689 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:47.815705    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp" podUID="734c6a7d-7e9d-4a9c-8577-263368974782" containerName="payload-pre-processing" containerID="cri-o://7e0eeece2348b8b34dca43e6bcf59e48d1fb95d655450761811a133a970b6423" gracePeriod=25
Nov 03 20:19:48.331827 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:48.324410    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp"
Nov 03 20:19:48.384058 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:48.384025    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/734c6a7d-7e9d-4a9c-8577-263368974782-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/734c6a7d-7e9d-4a9c-8577-263368974782-plugins-config-volume\") pod \"734c6a7d-7e9d-4a9c-8577-263368974782\" (UID: \"734c6a7d-7e9d-4a9c-8577-263368974782\") "
Nov 03 20:19:48.384262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:48.384095    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/734c6a7d-7e9d-4a9c-8577-263368974782-kube-api-access-sqx85\" (UniqueName: \"kubernetes.io/projected/734c6a7d-7e9d-4a9c-8577-263368974782-kube-api-access-sqx85\") pod \"734c6a7d-7e9d-4a9c-8577-263368974782\" (UID: \"734c6a7d-7e9d-4a9c-8577-263368974782\") "
Nov 03 20:19:48.385087 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:48.384754    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/734c6a7d-7e9d-4a9c-8577-263368974782-plugins-config-volume" pod "734c6a7d-7e9d-4a9c-8577-263368974782" (UID: "734c6a7d-7e9d-4a9c-8577-263368974782"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:19:48.385087 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:48.385057    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/734c6a7d-7e9d-4a9c-8577-263368974782-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:48.387443 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:48.387400    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734c6a7d-7e9d-4a9c-8577-263368974782-kube-api-access-sqx85" pod "734c6a7d-7e9d-4a9c-8577-263368974782" (UID: "734c6a7d-7e9d-4a9c-8577-263368974782"). InnerVolumeSpecName "kube-api-access-sqx85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:19:48.486942 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:48.486911    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqx85\" (UniqueName: \"kubernetes.io/projected/734c6a7d-7e9d-4a9c-8577-263368974782-kube-api-access-sqx85\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:19:48.772341 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:48.772299    4421 generic.go:356] "Generic (PLEG): container finished" podID="734c6a7d-7e9d-4a9c-8577-263368974782" containerID="7e0eeece2348b8b34dca43e6bcf59e48d1fb95d655450761811a133a970b6423" exitCode=0
Nov 03 20:19:48.773549 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:48.773527    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp"
Nov 03 20:19:48.773882 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:48.773666    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp" event={"ID":"734c6a7d-7e9d-4a9c-8577-263368974782","Type":"ContainerDied","Data":"7e0eeece2348b8b34dca43e6bcf59e48d1fb95d655450761811a133a970b6423"}
Nov 03 20:19:48.774148 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:48.774123    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp" event={"ID":"734c6a7d-7e9d-4a9c-8577-263368974782","Type":"ContainerDied","Data":"5ef59e34f8adb172bc78ca95b25fb502cfdf77472242c530c4e7957c915d938d"}
Nov 03 20:19:48.774312 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:48.774296    4421 scope.go:122] "RemoveContainer" containerID="7e0eeece2348b8b34dca43e6bcf59e48d1fb95d655450761811a133a970b6423"
Nov 03 20:19:48.785704 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:48.785684    4421 scope.go:122] "RemoveContainer" containerID="7e0eeece2348b8b34dca43e6bcf59e48d1fb95d655450761811a133a970b6423"
Nov 03 20:19:48.786057 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:19:48.786037    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e0eeece2348b8b34dca43e6bcf59e48d1fb95d655450761811a133a970b6423\": container with ID starting with 7e0eeece2348b8b34dca43e6bcf59e48d1fb95d655450761811a133a970b6423 not found: ID does not exist" containerID="7e0eeece2348b8b34dca43e6bcf59e48d1fb95d655450761811a133a970b6423"
Nov 03 20:19:48.786165 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:48.786064    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e0eeece2348b8b34dca43e6bcf59e48d1fb95d655450761811a133a970b6423"} err="failed to get container status \"7e0eeece2348b8b34dca43e6bcf59e48d1fb95d655450761811a133a970b6423\": rpc error: code = NotFound desc = could not find container \"7e0eeece2348b8b34dca43e6bcf59e48d1fb95d655450761811a133a970b6423\": container with ID starting with 7e0eeece2348b8b34dca43e6bcf59e48d1fb95d655450761811a133a970b6423 not found: ID does not exist"
Nov 03 20:19:48.804433 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:48.804392    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp"]
Nov 03 20:19:48.807781 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:48.807756    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9b65143e-7877bc9979-m7cdp"]
Nov 03 20:19:49.634105 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:19:49.634009    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:19:50.018775 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.018741    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="734c6a7d-7e9d-4a9c-8577-263368974782" path="/var/lib/kubelet/pods/734c6a7d-7e9d-4a9c-8577-263368974782/volumes"
Nov 03 20:19:50.288995 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.288916    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg"]
Nov 03 20:19:50.290656 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.290626    4421 cpu_manager.go:414] "containerMap: removing container" podUID="1a045c5d-d469-415e-8fc1-921f2ff05253" containerName="istio-proxy"
Nov 03 20:19:50.290798 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.290657    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a045c5d-d469-415e-8fc1-921f2ff05253" containerName="istio-proxy"
Nov 03 20:19:50.290798 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.290692    4421 cpu_manager.go:414] "containerMap: removing container" podUID="734c6a7d-7e9d-4a9c-8577-263368974782" containerName="payload-pre-processing"
Nov 03 20:19:50.290798 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.290702    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="734c6a7d-7e9d-4a9c-8577-263368974782" containerName="payload-pre-processing"
Nov 03 20:19:50.290904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.290872    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="734c6a7d-7e9d-4a9c-8577-263368974782" containerName="payload-pre-processing"
Nov 03 20:19:50.321284 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.321249    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c"]
Nov 03 20:19:50.322523 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.322498    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg"
Nov 03 20:19:50.335993 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.335965    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6b9849\""
Nov 03 20:19:50.349360 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.349331    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg"]
Nov 03 20:19:50.349480 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.349386    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c"]
Nov 03 20:19:50.349480 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.349422    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c"
Nov 03 20:19:50.354764 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.354745    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6b9849-dockercfg-86ctg\""
Nov 03 20:19:50.435628 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.430790    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvwp5\" (UniqueName: \"kubernetes.io/projected/ba4cba06-ec61-438f-ad5f-5b1145aa4789-kube-api-access-wvwp5\") pod \"payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c\" (UID: \"ba4cba06-ec61-438f-ad5f-5b1145aa4789\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c"
Nov 03 20:19:50.435628 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.430943    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba4cba06-ec61-438f-ad5f-5b1145aa4789-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c\" (UID: \"ba4cba06-ec61-438f-ad5f-5b1145aa4789\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c"
Nov 03 20:19:50.435628 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.430992    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e61dbc38-3815-4d2b-95ed-9b54f2cf0f54-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg\" (UID: \"e61dbc38-3815-4d2b-95ed-9b54f2cf0f54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg"
Nov 03 20:19:50.435628 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.431102    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b55j\" (UniqueName: \"kubernetes.io/projected/e61dbc38-3815-4d2b-95ed-9b54f2cf0f54-kube-api-access-4b55j\") pod \"payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg\" (UID: \"e61dbc38-3815-4d2b-95ed-9b54f2cf0f54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg"
Nov 03 20:19:50.582722 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.574672    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvwp5\" (UniqueName: \"kubernetes.io/projected/ba4cba06-ec61-438f-ad5f-5b1145aa4789-kube-api-access-wvwp5\") pod \"payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c\" (UID: \"ba4cba06-ec61-438f-ad5f-5b1145aa4789\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c"
Nov 03 20:19:50.582722 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.574812    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba4cba06-ec61-438f-ad5f-5b1145aa4789-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c\" (UID: \"ba4cba06-ec61-438f-ad5f-5b1145aa4789\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c"
Nov 03 20:19:50.582722 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.574855    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e61dbc38-3815-4d2b-95ed-9b54f2cf0f54-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg\" (UID: \"e61dbc38-3815-4d2b-95ed-9b54f2cf0f54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg"
Nov 03 20:19:50.582722 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.574941    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b55j\" (UniqueName: \"kubernetes.io/projected/e61dbc38-3815-4d2b-95ed-9b54f2cf0f54-kube-api-access-4b55j\") pod \"payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg\" (UID: \"e61dbc38-3815-4d2b-95ed-9b54f2cf0f54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg"
Nov 03 20:19:50.582722 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.576351    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba4cba06-ec61-438f-ad5f-5b1145aa4789-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c\" (UID: \"ba4cba06-ec61-438f-ad5f-5b1145aa4789\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c"
Nov 03 20:19:50.598647 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.597834    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e61dbc38-3815-4d2b-95ed-9b54f2cf0f54-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg\" (UID: \"e61dbc38-3815-4d2b-95ed-9b54f2cf0f54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg"
Nov 03 20:19:50.648622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.646186    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvwp5\" (UniqueName: \"kubernetes.io/projected/ba4cba06-ec61-438f-ad5f-5b1145aa4789-kube-api-access-wvwp5\") pod \"payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c\" (UID: \"ba4cba06-ec61-438f-ad5f-5b1145aa4789\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c"
Nov 03 20:19:50.648622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.646901    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b55j\" (UniqueName: \"kubernetes.io/projected/e61dbc38-3815-4d2b-95ed-9b54f2cf0f54-kube-api-access-4b55j\") pod \"payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg\" (UID: \"e61dbc38-3815-4d2b-95ed-9b54f2cf0f54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg"
Nov 03 20:19:50.663544 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.662838    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg"
Nov 03 20:19:50.691879 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:50.691545    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c"
Nov 03 20:19:51.037346 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:51.037309    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg"]
Nov 03 20:19:51.042161 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:51.041484    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c"]
Nov 03 20:19:51.240790 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:51.240745    4421 patch_prober.go:29] interesting pod/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:19:51.240962 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:51.240819    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln" podUID="b5488adb-f734-4d16-81bf-f230fca1306f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:51.836660 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:51.836615    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c" event={"ID":"ba4cba06-ec61-438f-ad5f-5b1145aa4789","Type":"ContainerStarted","Data":"7447bdabbfcec2a50756a4d9f8fa6fc16f6a72c6981b0741485788f5bff29082"}
Nov 03 20:19:51.836943 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:51.836708    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c" event={"ID":"ba4cba06-ec61-438f-ad5f-5b1145aa4789","Type":"ContainerStarted","Data":"bd2debfe3792bce650bee7394f6e9de110f71897ed5cfc6aab4ba9255f84a5da"}
Nov 03 20:19:51.836943 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:51.836769    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c"
Nov 03 20:19:51.839889 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:51.839795    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg" event={"ID":"e61dbc38-3815-4d2b-95ed-9b54f2cf0f54","Type":"ContainerStarted","Data":"c7589ae3ba3dba66e41df50b31cc034185ae85e5642159141eeb16d0f78d3a94"}
Nov 03 20:19:51.840069 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:51.839918    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg" event={"ID":"e61dbc38-3815-4d2b-95ed-9b54f2cf0f54","Type":"ContainerStarted","Data":"0ee2bafed218de13ca097264f3374c3089fbf39a8f6229de290e8c17dd50147e"}
Nov 03 20:19:51.840069 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:51.839980    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg"
Nov 03 20:19:51.860978 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:51.860919    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c" podStartSLOduration=1.86089518 podStartE2EDuration="1.86089518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:51.857248383 +0000 UTC m=+1121.990376845" watchObservedRunningTime="2026-11-03 20:19:51.86089518 +0000 UTC m=+1121.994023647"
Nov 03 20:19:51.880108 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:51.880060    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg" podStartSLOduration=1.880046809 podStartE2EDuration="1.880046809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:51.878060742 +0000 UTC m=+1122.011189205" watchObservedRunningTime="2026-11-03 20:19:51.880046809 +0000 UTC m=+1122.013175270"
Nov 03 20:19:52.342622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.327860    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"]
Nov 03 20:19:52.522641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.478431    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"]
Nov 03 20:19:52.522641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.478554    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.558070 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.556796    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-1b7b5e-openshift-default-dockercfg-7whx8\""
Nov 03 20:19:52.643752 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.635053    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-data\") pod \"e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") " pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.643752 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.635134    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-podinfo\") pod \"e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") " pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.643752 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.635167    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istiod-ca-cert\") pod \"e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") " pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.643752 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.635331    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-credential-socket\") pod \"e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") " pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.643752 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.635422    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcf8b\" (UniqueName: \"kubernetes.io/projected/9ffda400-707f-48d5-9e1f-7f76caf6cd58-kube-api-access-tcf8b\") pod \"e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") " pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.643752 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.635473    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-token\") pod \"e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") " pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.643752 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.635513    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-envoy\") pod \"e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") " pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.643752 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.635754    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-workload-socket\") pod \"e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") " pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.643752 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.635860    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-workload-certs\") pod \"e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") " pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.743635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.738907    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-podinfo\") pod \"e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") " pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.743635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.738983    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istiod-ca-cert\") pod \"e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") " pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.743635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.739079    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-credential-socket\") pod \"e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") " pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.743635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.739141    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcf8b\" (UniqueName: \"kubernetes.io/projected/9ffda400-707f-48d5-9e1f-7f76caf6cd58-kube-api-access-tcf8b\") pod \"e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") " pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.743635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.739194    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-token\") pod \"e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") " pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.743635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.739233    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-envoy\") pod \"e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") " pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.743635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.739369    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-workload-socket\") pod \"e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") " pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.743635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.739424    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-workload-certs\") pod \"e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") " pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.743635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.739531    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-data\") pod \"e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") " pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.743635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.740118    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-data\") pod \"e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") " pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.791970 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.791929    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istiod-ca-cert\") pod \"e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") " pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.792348 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.792319    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-credential-socket\") pod \"e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") " pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.793139 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.793107    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-workload-socket\") pod \"e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") " pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.794002 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.793977    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-workload-certs\") pod \"e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") " pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.796210 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.796188    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-podinfo\") pod \"e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") " pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.807208 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.807183    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcf8b\" (UniqueName: \"kubernetes.io/projected/9ffda400-707f-48d5-9e1f-7f76caf6cd58-kube-api-access-tcf8b\") pod \"e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") " pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.809352 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.809326    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-envoy\") pod \"e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") " pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.811006 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.810982    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-token\") pod \"e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") " pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:52.939298 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:52.939248    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:53.233653 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:19:53.233625    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ffda400_707f_48d5_9e1f_7f76caf6cd58.slice/crio-635e4f85666fcda5d78159e89395f69e8b71acd84e755366131e8b9aff7c1df4 WatchSource:0}: Error finding container 635e4f85666fcda5d78159e89395f69e8b71acd84e755366131e8b9aff7c1df4: Status 404 returned error can't find the container with id 635e4f85666fcda5d78159e89395f69e8b71acd84e755366131e8b9aff7c1df4
Nov 03 20:19:53.234367 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.234259    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"]
Nov 03 20:19:53.236524 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.236485    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:19:53.236632 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.236592    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:19:53.236692 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.236674    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:19:53.245267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.245243    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx"]
Nov 03 20:19:53.250995 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.250977    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx"
Nov 03 20:19:53.258060 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.257223    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-1127f5-dockercfg-bwghw\""
Nov 03 20:19:53.258060 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.257770    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-1127f5\""
Nov 03 20:19:53.261221 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.261201    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx"]
Nov 03 20:19:53.299277 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.299250    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g"]
Nov 03 20:19:53.307491 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.307467    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g"]
Nov 03 20:19:53.307704 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.307578    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g"
Nov 03 20:19:53.456166 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.456132    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8adedde-7209-409b-a289-aec02b4e6add-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g\" (UID: \"c8adedde-7209-409b-a289-aec02b4e6add\") " pod="openshift-ingress/payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g"
Nov 03 20:19:53.524126 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.524037    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7b664e-2b08-458d-bd6f-324ff24f8441-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx\" (UID: \"7f7b664e-2b08-458d-bd6f-324ff24f8441\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx"
Nov 03 20:19:53.524356 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.524335    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkrmj\" (UniqueName: \"kubernetes.io/projected/c8adedde-7209-409b-a289-aec02b4e6add-kube-api-access-tkrmj\") pod \"payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g\" (UID: \"c8adedde-7209-409b-a289-aec02b4e6add\") " pod="openshift-ingress/payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g"
Nov 03 20:19:53.524551 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.524532    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz5v6\" (UniqueName: \"kubernetes.io/projected/7f7b664e-2b08-458d-bd6f-324ff24f8441-kube-api-access-gz5v6\") pod \"payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx\" (UID: \"7f7b664e-2b08-458d-bd6f-324ff24f8441\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx"
Nov 03 20:19:53.639686 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.638623    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkrmj\" (UniqueName: \"kubernetes.io/projected/c8adedde-7209-409b-a289-aec02b4e6add-kube-api-access-tkrmj\") pod \"payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g\" (UID: \"c8adedde-7209-409b-a289-aec02b4e6add\") " pod="openshift-ingress/payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g"
Nov 03 20:19:53.643057 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.643031    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz5v6\" (UniqueName: \"kubernetes.io/projected/7f7b664e-2b08-458d-bd6f-324ff24f8441-kube-api-access-gz5v6\") pod \"payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx\" (UID: \"7f7b664e-2b08-458d-bd6f-324ff24f8441\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx"
Nov 03 20:19:53.645300 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.645276    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8adedde-7209-409b-a289-aec02b4e6add-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g\" (UID: \"c8adedde-7209-409b-a289-aec02b4e6add\") " pod="openshift-ingress/payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g"
Nov 03 20:19:53.645882 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.645848    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7b664e-2b08-458d-bd6f-324ff24f8441-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx\" (UID: \"7f7b664e-2b08-458d-bd6f-324ff24f8441\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx"
Nov 03 20:19:53.646445 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.646427    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8adedde-7209-409b-a289-aec02b4e6add-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g\" (UID: \"c8adedde-7209-409b-a289-aec02b4e6add\") " pod="openshift-ingress/payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g"
Nov 03 20:19:53.647312 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.647291    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7b664e-2b08-458d-bd6f-324ff24f8441-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx\" (UID: \"7f7b664e-2b08-458d-bd6f-324ff24f8441\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx"
Nov 03 20:19:53.653940 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.653917    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkrmj\" (UniqueName: \"kubernetes.io/projected/c8adedde-7209-409b-a289-aec02b4e6add-kube-api-access-tkrmj\") pod \"payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g\" (UID: \"c8adedde-7209-409b-a289-aec02b4e6add\") " pod="openshift-ingress/payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g"
Nov 03 20:19:53.657653 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.657635    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz5v6\" (UniqueName: \"kubernetes.io/projected/7f7b664e-2b08-458d-bd6f-324ff24f8441-kube-api-access-gz5v6\") pod \"payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx\" (UID: \"7f7b664e-2b08-458d-bd6f-324ff24f8441\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx"
Nov 03 20:19:53.858898 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.858802    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh" event={"ID":"9ffda400-707f-48d5-9e1f-7f76caf6cd58","Type":"ContainerStarted","Data":"d488539e315ffe818f2c07db4c61c67d6a4b3d20960e949dc1949477aae66ca8"}
Nov 03 20:19:53.859075 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.858916    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh" event={"ID":"9ffda400-707f-48d5-9e1f-7f76caf6cd58","Type":"ContainerStarted","Data":"635e4f85666fcda5d78159e89395f69e8b71acd84e755366131e8b9aff7c1df4"}
Nov 03 20:19:53.860872 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.860843    4421 generic.go:356] "Generic (PLEG): container finished" podID="b5488adb-f734-4d16-81bf-f230fca1306f" containerID="bd20d3285ea2dbd64059ed5b469e184320bfb51da263079d47d02b26fd6b69bd" exitCode=0
Nov 03 20:19:53.860992 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.860913    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln" event={"ID":"b5488adb-f734-4d16-81bf-f230fca1306f","Type":"ContainerDied","Data":"bd20d3285ea2dbd64059ed5b469e184320bfb51da263079d47d02b26fd6b69bd"}
Nov 03 20:19:53.861373 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.861358    4421 scope.go:122] "RemoveContainer" containerID="bd20d3285ea2dbd64059ed5b469e184320bfb51da263079d47d02b26fd6b69bd"
Nov 03 20:19:53.863015 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.862985    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:19:53.863132 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.863067    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:19:53.863132 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.863118    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:19:53.889864 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.889812    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh" podStartSLOduration=1.8897907840000001 podStartE2EDuration="1.889790784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:53.881679603 +0000 UTC m=+1124.014808068" watchObservedRunningTime="2026-11-03 20:19:53.889790784 +0000 UTC m=+1124.022919247"
Nov 03 20:19:53.899946 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.899918    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx"
Nov 03 20:19:53.933714 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.933688    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g"
Nov 03 20:19:53.941069 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:53.939409    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:54.389633 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:19:54.389569    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8adedde_7209_409b_a289_aec02b4e6add.slice/crio-423c86c771334a6bb89615861d6225feee8c75d7a54f47bacae600c9b2180145 WatchSource:0}: Error finding container 423c86c771334a6bb89615861d6225feee8c75d7a54f47bacae600c9b2180145: Status 404 returned error can't find the container with id 423c86c771334a6bb89615861d6225feee8c75d7a54f47bacae600c9b2180145
Nov 03 20:19:54.389947 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:54.389807    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g"]
Nov 03 20:19:54.429539 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:19:54.429516    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7b664e_2b08_458d_bd6f_324ff24f8441.slice/crio-1e10234d5b54c1364288b67923b444b79c44f7aa45789ed47ed06e0af01d9057 WatchSource:0}: Error finding container 1e10234d5b54c1364288b67923b444b79c44f7aa45789ed47ed06e0af01d9057: Status 404 returned error can't find the container with id 1e10234d5b54c1364288b67923b444b79c44f7aa45789ed47ed06e0af01d9057
Nov 03 20:19:54.429951 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:54.429888    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx"]
Nov 03 20:19:54.870683 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:54.870636    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx" event={"ID":"7f7b664e-2b08-458d-bd6f-324ff24f8441","Type":"ContainerStarted","Data":"a466a3757af60e6e4f9abc678ae234e4c5894b9af7d321088f4e34d4ed6379ef"}
Nov 03 20:19:54.871093 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:54.870718    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx"
Nov 03 20:19:54.871093 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:54.870746    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx" event={"ID":"7f7b664e-2b08-458d-bd6f-324ff24f8441","Type":"ContainerStarted","Data":"1e10234d5b54c1364288b67923b444b79c44f7aa45789ed47ed06e0af01d9057"}
Nov 03 20:19:54.872254 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:54.872231    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g" event={"ID":"c8adedde-7209-409b-a289-aec02b4e6add","Type":"ContainerStarted","Data":"8fc0aaf8d9ff4a44c57c62915771f13cf7561ce3d614486e56d758a86348fd6a"}
Nov 03 20:19:54.872402 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:54.872272    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g" event={"ID":"c8adedde-7209-409b-a289-aec02b4e6add","Type":"ContainerStarted","Data":"423c86c771334a6bb89615861d6225feee8c75d7a54f47bacae600c9b2180145"}
Nov 03 20:19:54.872402 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:54.872298    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g"
Nov 03 20:19:54.874338 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:54.874320    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln" event={"ID":"b5488adb-f734-4d16-81bf-f230fca1306f","Type":"ContainerStarted","Data":"4f64eed6da03a19d3a70975c168718c97c54c23aeefa8a6f64fd840dc7218325"}
Nov 03 20:19:54.889795 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:54.889739    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx" podStartSLOduration=1.889723241 podStartE2EDuration="1.889723241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:54.889330931 +0000 UTC m=+1125.022459394" watchObservedRunningTime="2026-11-03 20:19:54.889723241 +0000 UTC m=+1125.022851704"
Nov 03 20:19:54.893765 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:54.893740    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:19:54.909146 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:54.909106    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g" podStartSLOduration=1.909094304 podStartE2EDuration="1.909094304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:54.907252193 +0000 UTC m=+1125.040380657" watchObservedRunningTime="2026-11-03 20:19:54.909094304 +0000 UTC m=+1125.042222766"
Nov 03 20:19:54.945044 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:54.945022    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:55.891876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:55.891849    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:55.893918 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:55.893889    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:19:55.893918 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:55.893908    4421 patch_prober.go:29] interesting pod/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 20:19:55.894119 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:55.893960    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln" podUID="b5488adb-f734-4d16-81bf-f230fca1306f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 20:19:56.893878 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:56.893846    4421 patch_prober.go:29] interesting pod/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 20:19:56.894265 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:56.893896    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln" podUID="b5488adb-f734-4d16-81bf-f230fca1306f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 20:19:57.894062 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:57.894022    4421 patch_prober.go:29] interesting pod/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 20:19:57.894817 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:57.894778    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln" podUID="b5488adb-f734-4d16-81bf-f230fca1306f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 20:19:58.894549 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:58.894517    4421 patch_prober.go:29] interesting pod/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 20:19:58.894977 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:58.894615    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln" podUID="b5488adb-f734-4d16-81bf-f230fca1306f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 20:19:59.893940 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:59.893907    4421 patch_prober.go:29] interesting pod/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 20:19:59.894225 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:19:59.894200    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln" podUID="b5488adb-f734-4d16-81bf-f230fca1306f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 20:20:00.003267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:00.003234    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr"]
Nov 03 20:20:00.023081 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:00.023056    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr"
Nov 03 20:20:00.034161 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:00.034129    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1127f5-serving-cert\""
Nov 03 20:20:00.058645 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:00.040902    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr"]
Nov 03 20:20:00.216624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:00.191785    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvhbv\" (UniqueName: \"kubernetes.io/projected/0280365a-1ebd-4513-8cc6-ec821dc04960-kube-api-access-wvhbv\") pod \"maas-api-e2e-api-b-1127f5-776757c898-j42gr\" (UID: \"0280365a-1ebd-4513-8cc6-ec821dc04960\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr"
Nov 03 20:20:00.216624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:00.192016    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0280365a-1ebd-4513-8cc6-ec821dc04960-openshift-service-ca\") pod \"maas-api-e2e-api-b-1127f5-776757c898-j42gr\" (UID: \"0280365a-1ebd-4513-8cc6-ec821dc04960\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr"
Nov 03 20:20:00.216624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:00.192158    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0280365a-1ebd-4513-8cc6-ec821dc04960-maas-api-tls\") pod \"maas-api-e2e-api-b-1127f5-776757c898-j42gr\" (UID: \"0280365a-1ebd-4513-8cc6-ec821dc04960\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr"
Nov 03 20:20:00.216624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:00.192275    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0280365a-1ebd-4513-8cc6-ec821dc04960-metrics-certs\") pod \"maas-api-e2e-api-b-1127f5-776757c898-j42gr\" (UID: \"0280365a-1ebd-4513-8cc6-ec821dc04960\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr"
Nov 03 20:20:00.298739 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:00.298691    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0280365a-1ebd-4513-8cc6-ec821dc04960-maas-api-tls\") pod \"maas-api-e2e-api-b-1127f5-776757c898-j42gr\" (UID: \"0280365a-1ebd-4513-8cc6-ec821dc04960\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr"
Nov 03 20:20:00.299893 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:00.299867    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0280365a-1ebd-4513-8cc6-ec821dc04960-metrics-certs\") pod \"maas-api-e2e-api-b-1127f5-776757c898-j42gr\" (UID: \"0280365a-1ebd-4513-8cc6-ec821dc04960\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr"
Nov 03 20:20:00.300139 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:00.300122    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvhbv\" (UniqueName: \"kubernetes.io/projected/0280365a-1ebd-4513-8cc6-ec821dc04960-kube-api-access-wvhbv\") pod \"maas-api-e2e-api-b-1127f5-776757c898-j42gr\" (UID: \"0280365a-1ebd-4513-8cc6-ec821dc04960\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr"
Nov 03 20:20:00.300446 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:00.300428    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0280365a-1ebd-4513-8cc6-ec821dc04960-openshift-service-ca\") pod \"maas-api-e2e-api-b-1127f5-776757c898-j42gr\" (UID: \"0280365a-1ebd-4513-8cc6-ec821dc04960\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr"
Nov 03 20:20:00.302625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:00.301345    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0280365a-1ebd-4513-8cc6-ec821dc04960-openshift-service-ca\") pod \"maas-api-e2e-api-b-1127f5-776757c898-j42gr\" (UID: \"0280365a-1ebd-4513-8cc6-ec821dc04960\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr"
Nov 03 20:20:00.302734 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:00.302716    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0280365a-1ebd-4513-8cc6-ec821dc04960-maas-api-tls\") pod \"maas-api-e2e-api-b-1127f5-776757c898-j42gr\" (UID: \"0280365a-1ebd-4513-8cc6-ec821dc04960\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr"
Nov 03 20:20:00.304505 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:00.304447    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0280365a-1ebd-4513-8cc6-ec821dc04960-metrics-certs\") pod \"maas-api-e2e-api-b-1127f5-776757c898-j42gr\" (UID: \"0280365a-1ebd-4513-8cc6-ec821dc04960\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr"
Nov 03 20:20:00.312927 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:00.312897    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvhbv\" (UniqueName: \"kubernetes.io/projected/0280365a-1ebd-4513-8cc6-ec821dc04960-kube-api-access-wvhbv\") pod \"maas-api-e2e-api-b-1127f5-776757c898-j42gr\" (UID: \"0280365a-1ebd-4513-8cc6-ec821dc04960\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr"
Nov 03 20:20:00.404849 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:00.404818    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr"
Nov 03 20:20:00.718581 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:20:00.718547    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0280365a_1ebd_4513_8cc6_ec821dc04960.slice/crio-195f55df64ee23689dde2018cb737bce6ec92dadd3604518d417618e78c002df WatchSource:0}: Error finding container 195f55df64ee23689dde2018cb737bce6ec92dadd3604518d417618e78c002df: Status 404 returned error can't find the container with id 195f55df64ee23689dde2018cb737bce6ec92dadd3604518d417618e78c002df
Nov 03 20:20:00.720130 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:00.720106    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr"]
Nov 03 20:20:00.902107 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:00.900045    4421 patch_prober.go:29] interesting pod/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 20:20:00.902107 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:00.900117    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln" podUID="b5488adb-f734-4d16-81bf-f230fca1306f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 20:20:00.990350 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:00.990315    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr" event={"ID":"0280365a-1ebd-4513-8cc6-ec821dc04960","Type":"ContainerStarted","Data":"195f55df64ee23689dde2018cb737bce6ec92dadd3604518d417618e78c002df"}
Nov 03 20:20:01.898852 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:01.898821    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:20:01.899375 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:01.899345    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:20:01.900204 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:01.900182    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:20:02.004034 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:02.003983    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr" event={"ID":"0280365a-1ebd-4513-8cc6-ec821dc04960","Type":"ContainerStarted","Data":"272bb0d1ba70ce77de801d80bbeaac9a0436f74b241f5908634ef4bc6eb673c4"}
Nov 03 20:20:02.004231 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:02.004075    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr"
Nov 03 20:20:02.027331 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:02.027282    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr" podStartSLOduration=2.383151056 podStartE2EDuration="3.027267625s" totalImagesPullingTime="644.116569ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:02.024069937 +0000 UTC m=+1132.157198400" watchObservedRunningTime="2026-11-03 20:20:02.027267625 +0000 UTC m=+1132.160396125"
Nov 03 20:20:02.851152 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:02.851122    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c"
Nov 03 20:20:02.851325 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:02.851206    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg"
Nov 03 20:20:05.907636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:05.895710    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g"
Nov 03 20:20:05.981625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:05.980990    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx"
Nov 03 20:20:08.041145 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:08.041120    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr"
Nov 03 20:20:08.984372 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:08.984342    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c"]
Nov 03 20:20:09.010345 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:09.006788    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg"]
Nov 03 20:20:11.706807 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:11.706747    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6"]
Nov 03 20:20:11.714151 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:11.714125    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6"
Nov 03 20:20:11.725455 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:11.724789    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6"]
Nov 03 20:20:11.725455 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:11.725028    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9b65143e-serving-cert\""
Nov 03 20:20:11.772420 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:11.772384    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1e67982-68de-4503-af48-8cb554c6679e-maas-api-tls\") pod \"maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6\" (UID: \"f1e67982-68de-4503-af48-8cb554c6679e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6"
Nov 03 20:20:11.772558 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:11.772451    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1e67982-68de-4503-af48-8cb554c6679e-openshift-service-ca\") pod \"maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6\" (UID: \"f1e67982-68de-4503-af48-8cb554c6679e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6"
Nov 03 20:20:11.772717 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:11.772691    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfwtj\" (UniqueName: \"kubernetes.io/projected/f1e67982-68de-4503-af48-8cb554c6679e-kube-api-access-jfwtj\") pod \"maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6\" (UID: \"f1e67982-68de-4503-af48-8cb554c6679e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6"
Nov 03 20:20:11.772888 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:11.772872    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1e67982-68de-4503-af48-8cb554c6679e-metrics-certs\") pod \"maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6\" (UID: \"f1e67982-68de-4503-af48-8cb554c6679e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6"
Nov 03 20:20:11.874304 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:11.874266    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1e67982-68de-4503-af48-8cb554c6679e-metrics-certs\") pod \"maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6\" (UID: \"f1e67982-68de-4503-af48-8cb554c6679e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6"
Nov 03 20:20:11.874477 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:11.874381    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1e67982-68de-4503-af48-8cb554c6679e-maas-api-tls\") pod \"maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6\" (UID: \"f1e67982-68de-4503-af48-8cb554c6679e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6"
Nov 03 20:20:11.874477 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:11.874421    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1e67982-68de-4503-af48-8cb554c6679e-openshift-service-ca\") pod \"maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6\" (UID: \"f1e67982-68de-4503-af48-8cb554c6679e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6"
Nov 03 20:20:11.874588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:11.874565    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfwtj\" (UniqueName: \"kubernetes.io/projected/f1e67982-68de-4503-af48-8cb554c6679e-kube-api-access-jfwtj\") pod \"maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6\" (UID: \"f1e67982-68de-4503-af48-8cb554c6679e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6"
Nov 03 20:20:11.875224 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:11.875148    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1e67982-68de-4503-af48-8cb554c6679e-openshift-service-ca\") pod \"maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6\" (UID: \"f1e67982-68de-4503-af48-8cb554c6679e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6"
Nov 03 20:20:11.877065 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:11.877045    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1e67982-68de-4503-af48-8cb554c6679e-maas-api-tls\") pod \"maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6\" (UID: \"f1e67982-68de-4503-af48-8cb554c6679e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6"
Nov 03 20:20:11.877168 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:11.877090    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1e67982-68de-4503-af48-8cb554c6679e-metrics-certs\") pod \"maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6\" (UID: \"f1e67982-68de-4503-af48-8cb554c6679e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6"
Nov 03 20:20:11.887120 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:11.887098    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfwtj\" (UniqueName: \"kubernetes.io/projected/f1e67982-68de-4503-af48-8cb554c6679e-kube-api-access-jfwtj\") pod \"maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6\" (UID: \"f1e67982-68de-4503-af48-8cb554c6679e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6"
Nov 03 20:20:12.048684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:12.042016    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6"
Nov 03 20:20:12.486198 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:12.486168    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6"]
Nov 03 20:20:12.495482 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:20:12.495453    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e67982_68de_4503_af48_8cb554c6679e.slice/crio-c8401976e66aef3d1b5912cda01dabbeefcfbda4f1d13810e47837276d7281e8 WatchSource:0}: Error finding container c8401976e66aef3d1b5912cda01dabbeefcfbda4f1d13810e47837276d7281e8: Status 404 returned error can't find the container with id c8401976e66aef3d1b5912cda01dabbeefcfbda4f1d13810e47837276d7281e8
Nov 03 20:20:13.298414 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:13.298377    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6" event={"ID":"f1e67982-68de-4503-af48-8cb554c6679e","Type":"ContainerStarted","Data":"d590806b4d3b0ddd26d3ec0025ce3fb0416e1db4206f676d107d24c3a73c9a8f"}
Nov 03 20:20:13.299017 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:13.298445    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6" event={"ID":"f1e67982-68de-4503-af48-8cb554c6679e","Type":"ContainerStarted","Data":"c8401976e66aef3d1b5912cda01dabbeefcfbda4f1d13810e47837276d7281e8"}
Nov 03 20:20:13.299017 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:13.298467    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6"
Nov 03 20:20:13.322423 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:13.322336    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6" podStartSLOduration=1.765343123 podStartE2EDuration="2.322322137s" totalImagesPullingTime="556.979014ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:13.318096296 +0000 UTC m=+1143.451224759" watchObservedRunningTime="2026-11-03 20:20:13.322322137 +0000 UTC m=+1143.455450913"
Nov 03 20:20:13.985121 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:13.985081    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c" podUID="ba4cba06-ec61-438f-ad5f-5b1145aa4789" containerName="payload-processing" containerID="cri-o://7447bdabbfcec2a50756a4d9f8fa6fc16f6a72c6981b0741485788f5bff29082" gracePeriod=25
Nov 03 20:20:14.007523 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:14.007494    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg" podUID="e61dbc38-3815-4d2b-95ed-9b54f2cf0f54" containerName="payload-pre-processing" containerID="cri-o://c7589ae3ba3dba66e41df50b31cc034185ae85e5642159141eeb16d0f78d3a94" gracePeriod=25
Nov 03 20:20:14.437694 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:14.437571    4421 generic.go:356] "Generic (PLEG): container finished" podID="ba4cba06-ec61-438f-ad5f-5b1145aa4789" containerID="7447bdabbfcec2a50756a4d9f8fa6fc16f6a72c6981b0741485788f5bff29082" exitCode=0
Nov 03 20:20:14.438121 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:14.437811    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c" event={"ID":"ba4cba06-ec61-438f-ad5f-5b1145aa4789","Type":"ContainerDied","Data":"7447bdabbfcec2a50756a4d9f8fa6fc16f6a72c6981b0741485788f5bff29082"}
Nov 03 20:20:14.442489 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:14.442462    4421 generic.go:356] "Generic (PLEG): container finished" podID="e61dbc38-3815-4d2b-95ed-9b54f2cf0f54" containerID="c7589ae3ba3dba66e41df50b31cc034185ae85e5642159141eeb16d0f78d3a94" exitCode=0
Nov 03 20:20:14.443715 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:14.443691    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg" event={"ID":"e61dbc38-3815-4d2b-95ed-9b54f2cf0f54","Type":"ContainerDied","Data":"c7589ae3ba3dba66e41df50b31cc034185ae85e5642159141eeb16d0f78d3a94"}
Nov 03 20:20:14.449294 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:14.448383    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c"
Nov 03 20:20:14.451940 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:14.451864    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg"
Nov 03 20:20:14.514259 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:14.514180    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba4cba06-ec61-438f-ad5f-5b1145aa4789-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba4cba06-ec61-438f-ad5f-5b1145aa4789-plugins-config-volume\") pod \"ba4cba06-ec61-438f-ad5f-5b1145aa4789\" (UID: \"ba4cba06-ec61-438f-ad5f-5b1145aa4789\") "
Nov 03 20:20:14.514397 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:14.514307    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e61dbc38-3815-4d2b-95ed-9b54f2cf0f54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e61dbc38-3815-4d2b-95ed-9b54f2cf0f54-plugins-config-volume\") pod \"e61dbc38-3815-4d2b-95ed-9b54f2cf0f54\" (UID: \"e61dbc38-3815-4d2b-95ed-9b54f2cf0f54\") "
Nov 03 20:20:14.514457 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:14.514409    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e61dbc38-3815-4d2b-95ed-9b54f2cf0f54-kube-api-access-4b55j\" (UniqueName: \"kubernetes.io/projected/e61dbc38-3815-4d2b-95ed-9b54f2cf0f54-kube-api-access-4b55j\") pod \"e61dbc38-3815-4d2b-95ed-9b54f2cf0f54\" (UID: \"e61dbc38-3815-4d2b-95ed-9b54f2cf0f54\") "
Nov 03 20:20:14.514457 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:14.514449    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba4cba06-ec61-438f-ad5f-5b1145aa4789-kube-api-access-wvwp5\" (UniqueName: \"kubernetes.io/projected/ba4cba06-ec61-438f-ad5f-5b1145aa4789-kube-api-access-wvwp5\") pod \"ba4cba06-ec61-438f-ad5f-5b1145aa4789\" (UID: \"ba4cba06-ec61-438f-ad5f-5b1145aa4789\") "
Nov 03 20:20:14.514685 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:14.514653    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba4cba06-ec61-438f-ad5f-5b1145aa4789-plugins-config-volume" pod "ba4cba06-ec61-438f-ad5f-5b1145aa4789" (UID: "ba4cba06-ec61-438f-ad5f-5b1145aa4789"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:14.515449 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:14.514768    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e61dbc38-3815-4d2b-95ed-9b54f2cf0f54-plugins-config-volume" pod "e61dbc38-3815-4d2b-95ed-9b54f2cf0f54" (UID: "e61dbc38-3815-4d2b-95ed-9b54f2cf0f54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:14.515907 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:14.515881    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba4cba06-ec61-438f-ad5f-5b1145aa4789-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:14.515997 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:14.515910    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e61dbc38-3815-4d2b-95ed-9b54f2cf0f54-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:14.517843 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:14.517816    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba4cba06-ec61-438f-ad5f-5b1145aa4789-kube-api-access-wvwp5" pod "ba4cba06-ec61-438f-ad5f-5b1145aa4789" (UID: "ba4cba06-ec61-438f-ad5f-5b1145aa4789"). InnerVolumeSpecName "kube-api-access-wvwp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:14.517948 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:14.517849    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61dbc38-3815-4d2b-95ed-9b54f2cf0f54-kube-api-access-4b55j" pod "e61dbc38-3815-4d2b-95ed-9b54f2cf0f54" (UID: "e61dbc38-3815-4d2b-95ed-9b54f2cf0f54"). InnerVolumeSpecName "kube-api-access-4b55j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:14.617115 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:14.617068    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b55j\" (UniqueName: \"kubernetes.io/projected/e61dbc38-3815-4d2b-95ed-9b54f2cf0f54-kube-api-access-4b55j\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:14.617115 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:14.617102    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvwp5\" (UniqueName: \"kubernetes.io/projected/ba4cba06-ec61-438f-ad5f-5b1145aa4789-kube-api-access-wvwp5\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:15.451075 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:15.451035    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c" event={"ID":"ba4cba06-ec61-438f-ad5f-5b1145aa4789","Type":"ContainerDied","Data":"bd2debfe3792bce650bee7394f6e9de110f71897ed5cfc6aab4ba9255f84a5da"}
Nov 03 20:20:15.451075 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:15.451066    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c"
Nov 03 20:20:15.451573 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:15.451104    4421 scope.go:122] "RemoveContainer" containerID="7447bdabbfcec2a50756a4d9f8fa6fc16f6a72c6981b0741485788f5bff29082"
Nov 03 20:20:15.452529 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:15.452510    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg"
Nov 03 20:20:15.452631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:15.452527    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg" event={"ID":"e61dbc38-3815-4d2b-95ed-9b54f2cf0f54","Type":"ContainerDied","Data":"0ee2bafed218de13ca097264f3374c3089fbf39a8f6229de290e8c17dd50147e"}
Nov 03 20:20:15.465741 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:15.463691    4421 scope.go:122] "RemoveContainer" containerID="c7589ae3ba3dba66e41df50b31cc034185ae85e5642159141eeb16d0f78d3a94"
Nov 03 20:20:15.483677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:15.483648    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg"]
Nov 03 20:20:15.488299 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:15.488278    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b9849-85fb55f8c5-6qbwg"]
Nov 03 20:20:15.499206 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:15.499178    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c"]
Nov 03 20:20:15.503246 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:15.503212    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b9849-866bfd4988-rqp4c"]
Nov 03 20:20:16.071274 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:16.071123    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba4cba06-ec61-438f-ad5f-5b1145aa4789" path="/var/lib/kubelet/pods/ba4cba06-ec61-438f-ad5f-5b1145aa4789/volumes"
Nov 03 20:20:16.071855 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:16.071819    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e61dbc38-3815-4d2b-95ed-9b54f2cf0f54" path="/var/lib/kubelet/pods/e61dbc38-3815-4d2b-95ed-9b54f2cf0f54/volumes"
Nov 03 20:20:18.905621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:18.903971    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk"]
Nov 03 20:20:18.927623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:18.927563    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5"]
Nov 03 20:20:19.490859 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:19.490825    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6"
Nov 03 20:20:19.732365 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:20:19.732329    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:20:23.908622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:23.907624    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk" podUID="a1903a77-2443-4c41-984c-45b71e22201a" containerName="payload-processing" containerID="cri-o://a687832d0a0144ee51c273d964e2318447f620f1a601800c74fbd98171abcd65" gracePeriod=25
Nov 03 20:20:23.934620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:23.930668    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5" podUID="a087d02d-46f7-4eb1-b085-52f542894f70" containerName="payload-pre-processing" containerID="cri-o://77f6c28ffcc2e0e7e35957097d14b2488f453351c83127789d6c30bdcc3e5128" gracePeriod=25
Nov 03 20:20:24.689335 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:24.689304    4421 generic.go:356] "Generic (PLEG): container finished" podID="a087d02d-46f7-4eb1-b085-52f542894f70" containerID="77f6c28ffcc2e0e7e35957097d14b2488f453351c83127789d6c30bdcc3e5128" exitCode=0
Nov 03 20:20:24.689494 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:24.689433    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5" event={"ID":"a087d02d-46f7-4eb1-b085-52f542894f70","Type":"ContainerDied","Data":"77f6c28ffcc2e0e7e35957097d14b2488f453351c83127789d6c30bdcc3e5128"}
Nov 03 20:20:24.693321 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:24.693289    4421 generic.go:356] "Generic (PLEG): container finished" podID="a1903a77-2443-4c41-984c-45b71e22201a" containerID="a687832d0a0144ee51c273d964e2318447f620f1a601800c74fbd98171abcd65" exitCode=0
Nov 03 20:20:24.693478 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:24.693464    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk" event={"ID":"a1903a77-2443-4c41-984c-45b71e22201a","Type":"ContainerDied","Data":"a687832d0a0144ee51c273d964e2318447f620f1a601800c74fbd98171abcd65"}
Nov 03 20:20:24.751658 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:24.750996    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5"
Nov 03 20:20:24.839376 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:24.839242    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a087d02d-46f7-4eb1-b085-52f542894f70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a087d02d-46f7-4eb1-b085-52f542894f70-plugins-config-volume\") pod \"a087d02d-46f7-4eb1-b085-52f542894f70\" (UID: \"a087d02d-46f7-4eb1-b085-52f542894f70\") "
Nov 03 20:20:24.839376 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:24.839302    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a087d02d-46f7-4eb1-b085-52f542894f70-kube-api-access-zmdvn\" (UniqueName: \"kubernetes.io/projected/a087d02d-46f7-4eb1-b085-52f542894f70-kube-api-access-zmdvn\") pod \"a087d02d-46f7-4eb1-b085-52f542894f70\" (UID: \"a087d02d-46f7-4eb1-b085-52f542894f70\") "
Nov 03 20:20:24.860620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:24.858624    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a087d02d-46f7-4eb1-b085-52f542894f70-plugins-config-volume" pod "a087d02d-46f7-4eb1-b085-52f542894f70" (UID: "a087d02d-46f7-4eb1-b085-52f542894f70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:24.867837 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:24.866392    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a087d02d-46f7-4eb1-b085-52f542894f70-kube-api-access-zmdvn" pod "a087d02d-46f7-4eb1-b085-52f542894f70" (UID: "a087d02d-46f7-4eb1-b085-52f542894f70"). InnerVolumeSpecName "kube-api-access-zmdvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:24.881681 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:24.881341    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk"
Nov 03 20:20:24.958189 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:24.958153    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1903a77-2443-4c41-984c-45b71e22201a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1903a77-2443-4c41-984c-45b71e22201a-plugins-config-volume\") pod \"a1903a77-2443-4c41-984c-45b71e22201a\" (UID: \"a1903a77-2443-4c41-984c-45b71e22201a\") "
Nov 03 20:20:24.959461 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:24.958289    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1903a77-2443-4c41-984c-45b71e22201a-kube-api-access-kgrq9\" (UniqueName: \"kubernetes.io/projected/a1903a77-2443-4c41-984c-45b71e22201a-kube-api-access-kgrq9\") pod \"a1903a77-2443-4c41-984c-45b71e22201a\" (UID: \"a1903a77-2443-4c41-984c-45b71e22201a\") "
Nov 03 20:20:24.959461 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:24.959001    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1903a77-2443-4c41-984c-45b71e22201a-plugins-config-volume" pod "a1903a77-2443-4c41-984c-45b71e22201a" (UID: "a1903a77-2443-4c41-984c-45b71e22201a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:24.959461 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:24.959247    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a087d02d-46f7-4eb1-b085-52f542894f70-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:24.959461 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:24.959268    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmdvn\" (UniqueName: \"kubernetes.io/projected/a087d02d-46f7-4eb1-b085-52f542894f70-kube-api-access-zmdvn\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:24.959461 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:24.959283    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1903a77-2443-4c41-984c-45b71e22201a-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:24.960922 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:24.960894    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1903a77-2443-4c41-984c-45b71e22201a-kube-api-access-kgrq9" pod "a1903a77-2443-4c41-984c-45b71e22201a" (UID: "a1903a77-2443-4c41-984c-45b71e22201a"). InnerVolumeSpecName "kube-api-access-kgrq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:25.060505 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:25.060458    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgrq9\" (UniqueName: \"kubernetes.io/projected/a1903a77-2443-4c41-984c-45b71e22201a-kube-api-access-kgrq9\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:25.704440 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:25.704414    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5"
Nov 03 20:20:25.704682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:25.704444    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5" event={"ID":"a087d02d-46f7-4eb1-b085-52f542894f70","Type":"ContainerDied","Data":"65f10ca2fcd1096999458c2ed4cb3de8239d899b5dd9b8c0c83a0d95299053fc"}
Nov 03 20:20:25.704682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:25.704523    4421 scope.go:122] "RemoveContainer" containerID="77f6c28ffcc2e0e7e35957097d14b2488f453351c83127789d6c30bdcc3e5128"
Nov 03 20:20:25.706169 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:25.706081    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk"
Nov 03 20:20:25.706169 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:25.706092    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk" event={"ID":"a1903a77-2443-4c41-984c-45b71e22201a","Type":"ContainerDied","Data":"5f855635fc08d1f3480b77850af5b001d9fbc893030ca0bc9c8d92c31b025997"}
Nov 03 20:20:25.719734 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:25.719692    4421 scope.go:122] "RemoveContainer" containerID="a687832d0a0144ee51c273d964e2318447f620f1a601800c74fbd98171abcd65"
Nov 03 20:20:25.734541 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:25.734520    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5"]
Nov 03 20:20:25.740334 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:25.740307    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9a9ea9-56fbd555c5-wnsb5"]
Nov 03 20:20:25.748833 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:25.748799    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk"]
Nov 03 20:20:25.752818 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:25.752800    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9a9ea9-5d69fc846b-7bfhk"]
Nov 03 20:20:26.020619 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:26.020489    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a087d02d-46f7-4eb1-b085-52f542894f70" path="/var/lib/kubelet/pods/a087d02d-46f7-4eb1-b085-52f542894f70/volumes"
Nov 03 20:20:26.021244 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:26.021211    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1903a77-2443-4c41-984c-45b71e22201a" path="/var/lib/kubelet/pods/a1903a77-2443-4c41-984c-45b71e22201a/volumes"
Nov 03 20:20:26.932799 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:26.932751    4421 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:20:26.933259 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:26.932821    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh" podUID="9ffda400-707f-48d5-9e1f-7f76caf6cd58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:27.679541 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:27.679508    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj"]
Nov 03 20:20:27.680168 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:27.680154    4421 cpu_manager.go:414] "containerMap: removing container" podUID="a1903a77-2443-4c41-984c-45b71e22201a" containerName="payload-processing"
Nov 03 20:20:27.680213 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:27.680169    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1903a77-2443-4c41-984c-45b71e22201a" containerName="payload-processing"
Nov 03 20:20:27.680213 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:27.680186    4421 cpu_manager.go:414] "containerMap: removing container" podUID="a087d02d-46f7-4eb1-b085-52f542894f70" containerName="payload-pre-processing"
Nov 03 20:20:27.680213 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:27.680191    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a087d02d-46f7-4eb1-b085-52f542894f70" containerName="payload-pre-processing"
Nov 03 20:20:27.680213 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:27.680211    4421 cpu_manager.go:414] "containerMap: removing container" podUID="e61dbc38-3815-4d2b-95ed-9b54f2cf0f54" containerName="payload-pre-processing"
Nov 03 20:20:27.680351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:27.680217    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e61dbc38-3815-4d2b-95ed-9b54f2cf0f54" containerName="payload-pre-processing"
Nov 03 20:20:27.680351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:27.680237    4421 cpu_manager.go:414] "containerMap: removing container" podUID="ba4cba06-ec61-438f-ad5f-5b1145aa4789" containerName="payload-processing"
Nov 03 20:20:27.680351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:27.680243    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba4cba06-ec61-438f-ad5f-5b1145aa4789" containerName="payload-processing"
Nov 03 20:20:27.680351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:27.680321    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a087d02d-46f7-4eb1-b085-52f542894f70" containerName="payload-pre-processing"
Nov 03 20:20:27.680351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:27.680335    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e61dbc38-3815-4d2b-95ed-9b54f2cf0f54" containerName="payload-pre-processing"
Nov 03 20:20:27.680351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:27.680345    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba4cba06-ec61-438f-ad5f-5b1145aa4789" containerName="payload-processing"
Nov 03 20:20:27.680351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:27.680353    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1903a77-2443-4c41-984c-45b71e22201a" containerName="payload-processing"
Nov 03 20:20:27.846623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:27.826477    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj"]
Nov 03 20:20:27.846623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:27.826534    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg"]
Nov 03 20:20:27.860296 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:27.847027    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj"
Nov 03 20:20:27.933671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:27.933356    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg"]
Nov 03 20:20:27.933671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:27.933491    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg"
Nov 03 20:20:27.955365 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:27.954737    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1b7b5e-dockercfg-d8vbz\""
Nov 03 20:20:27.955365 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:27.954855    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1b7b5e\""
Nov 03 20:20:27.968528 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:27.963677    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htstd\" (UniqueName: \"kubernetes.io/projected/3ab7a061-3614-41d6-af5a-25dde57e6bbd-kube-api-access-htstd\") pod \"payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj\" (UID: \"3ab7a061-3614-41d6-af5a-25dde57e6bbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj"
Nov 03 20:20:27.968528 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:27.964079    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ab7a061-3614-41d6-af5a-25dde57e6bbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj\" (UID: \"3ab7a061-3614-41d6-af5a-25dde57e6bbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj"
Nov 03 20:20:28.080666 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:28.080633    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9259c57-03ab-490b-bbde-f40be00f9ad4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg\" (UID: \"f9259c57-03ab-490b-bbde-f40be00f9ad4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg"
Nov 03 20:20:28.080989 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:28.080970    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4sw8\" (UniqueName: \"kubernetes.io/projected/f9259c57-03ab-490b-bbde-f40be00f9ad4-kube-api-access-d4sw8\") pod \"payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg\" (UID: \"f9259c57-03ab-490b-bbde-f40be00f9ad4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg"
Nov 03 20:20:28.081180 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:28.081163    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htstd\" (UniqueName: \"kubernetes.io/projected/3ab7a061-3614-41d6-af5a-25dde57e6bbd-kube-api-access-htstd\") pod \"payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj\" (UID: \"3ab7a061-3614-41d6-af5a-25dde57e6bbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj"
Nov 03 20:20:28.081909 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:28.081886    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ab7a061-3614-41d6-af5a-25dde57e6bbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj\" (UID: \"3ab7a061-3614-41d6-af5a-25dde57e6bbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj"
Nov 03 20:20:28.083025 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:28.082866    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ab7a061-3614-41d6-af5a-25dde57e6bbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj\" (UID: \"3ab7a061-3614-41d6-af5a-25dde57e6bbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj"
Nov 03 20:20:28.094715 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:28.094687    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htstd\" (UniqueName: \"kubernetes.io/projected/3ab7a061-3614-41d6-af5a-25dde57e6bbd-kube-api-access-htstd\") pod \"payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj\" (UID: \"3ab7a061-3614-41d6-af5a-25dde57e6bbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj"
Nov 03 20:20:28.184249 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:28.184148    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9259c57-03ab-490b-bbde-f40be00f9ad4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg\" (UID: \"f9259c57-03ab-490b-bbde-f40be00f9ad4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg"
Nov 03 20:20:28.184249 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:28.184204    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4sw8\" (UniqueName: \"kubernetes.io/projected/f9259c57-03ab-490b-bbde-f40be00f9ad4-kube-api-access-d4sw8\") pod \"payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg\" (UID: \"f9259c57-03ab-490b-bbde-f40be00f9ad4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg"
Nov 03 20:20:28.184967 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:28.184941    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9259c57-03ab-490b-bbde-f40be00f9ad4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg\" (UID: \"f9259c57-03ab-490b-bbde-f40be00f9ad4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg"
Nov 03 20:20:28.196729 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:28.196709    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4sw8\" (UniqueName: \"kubernetes.io/projected/f9259c57-03ab-490b-bbde-f40be00f9ad4-kube-api-access-d4sw8\") pod \"payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg\" (UID: \"f9259c57-03ab-490b-bbde-f40be00f9ad4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg"
Nov 03 20:20:28.324695 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:28.324663    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg"
Nov 03 20:20:28.394911 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:28.394826    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj"
Nov 03 20:20:28.703003 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:20:28.702972    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9259c57_03ab_490b_bbde_f40be00f9ad4.slice/crio-3b5fcfb8aa894696f16bd5efd31289ee338306bb11340d16d8e2ede1eaa95461 WatchSource:0}: Error finding container 3b5fcfb8aa894696f16bd5efd31289ee338306bb11340d16d8e2ede1eaa95461: Status 404 returned error can't find the container with id 3b5fcfb8aa894696f16bd5efd31289ee338306bb11340d16d8e2ede1eaa95461
Nov 03 20:20:28.703427 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:28.703373    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg"]
Nov 03 20:20:28.733360 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:20:28.733331    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab7a061_3614_41d6_af5a_25dde57e6bbd.slice/crio-5eb6a3844e411a59557f38db0ba435c48d3b14bb5d8ff43b4d38b159fd40a79c WatchSource:0}: Error finding container 5eb6a3844e411a59557f38db0ba435c48d3b14bb5d8ff43b4d38b159fd40a79c: Status 404 returned error can't find the container with id 5eb6a3844e411a59557f38db0ba435c48d3b14bb5d8ff43b4d38b159fd40a79c
Nov 03 20:20:28.737357 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:28.735397    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj"]
Nov 03 20:20:28.744455 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:28.744418    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg" event={"ID":"f9259c57-03ab-490b-bbde-f40be00f9ad4","Type":"ContainerStarted","Data":"3b5fcfb8aa894696f16bd5efd31289ee338306bb11340d16d8e2ede1eaa95461"}
Nov 03 20:20:29.759675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:29.759588    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj" event={"ID":"3ab7a061-3614-41d6-af5a-25dde57e6bbd","Type":"ContainerStarted","Data":"5eb6a3844e411a59557f38db0ba435c48d3b14bb5d8ff43b4d38b159fd40a79c"}
Nov 03 20:20:29.837927 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:29.837884    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn"]
Nov 03 20:20:29.838572 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:29.838292    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn" podUID="6c9e5419-faf0-42aa-befb-79149b3cd0f1" containerName="maas-api" containerID="cri-o://94ba640b5d22ebb632c7aa9f78d29859ba6759126c2f7569d33f6d21c67aa315" gracePeriod=30
Nov 03 20:20:30.421785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:30.421751    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn"
Nov 03 20:20:31.237904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:30.519352    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c9e5419-faf0-42aa-befb-79149b3cd0f1-kube-api-access-x4lfs\" (UniqueName: \"kubernetes.io/projected/6c9e5419-faf0-42aa-befb-79149b3cd0f1-kube-api-access-x4lfs\") pod \"6c9e5419-faf0-42aa-befb-79149b3cd0f1\" (UID: \"6c9e5419-faf0-42aa-befb-79149b3cd0f1\") "
Nov 03 20:20:31.237904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:30.519420    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c9e5419-faf0-42aa-befb-79149b3cd0f1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9e5419-faf0-42aa-befb-79149b3cd0f1-maas-api-tls\") pod \"6c9e5419-faf0-42aa-befb-79149b3cd0f1\" (UID: \"6c9e5419-faf0-42aa-befb-79149b3cd0f1\") "
Nov 03 20:20:31.237904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:30.519494    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c9e5419-faf0-42aa-befb-79149b3cd0f1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9e5419-faf0-42aa-befb-79149b3cd0f1-metrics-certs\") pod \"6c9e5419-faf0-42aa-befb-79149b3cd0f1\" (UID: \"6c9e5419-faf0-42aa-befb-79149b3cd0f1\") "
Nov 03 20:20:31.237904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:30.519589    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c9e5419-faf0-42aa-befb-79149b3cd0f1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c9e5419-faf0-42aa-befb-79149b3cd0f1-openshift-service-ca\") pod \"6c9e5419-faf0-42aa-befb-79149b3cd0f1\" (UID: \"6c9e5419-faf0-42aa-befb-79149b3cd0f1\") "
Nov 03 20:20:31.237904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:30.520478    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c9e5419-faf0-42aa-befb-79149b3cd0f1-openshift-service-ca" pod "6c9e5419-faf0-42aa-befb-79149b3cd0f1" (UID: "6c9e5419-faf0-42aa-befb-79149b3cd0f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:31.237904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:30.522511    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9e5419-faf0-42aa-befb-79149b3cd0f1-kube-api-access-x4lfs" pod "6c9e5419-faf0-42aa-befb-79149b3cd0f1" (UID: "6c9e5419-faf0-42aa-befb-79149b3cd0f1"). InnerVolumeSpecName "kube-api-access-x4lfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:31.237904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:30.522806    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c9e5419-faf0-42aa-befb-79149b3cd0f1-metrics-certs" pod "6c9e5419-faf0-42aa-befb-79149b3cd0f1" (UID: "6c9e5419-faf0-42aa-befb-79149b3cd0f1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:31.237904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:30.526800    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c9e5419-faf0-42aa-befb-79149b3cd0f1-maas-api-tls" pod "6c9e5419-faf0-42aa-befb-79149b3cd0f1" (UID: "6c9e5419-faf0-42aa-befb-79149b3cd0f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:31.237904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:30.627320    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4lfs\" (UniqueName: \"kubernetes.io/projected/6c9e5419-faf0-42aa-befb-79149b3cd0f1-kube-api-access-x4lfs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:31.237904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:30.627348    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9e5419-faf0-42aa-befb-79149b3cd0f1-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:31.237904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:30.627365    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9e5419-faf0-42aa-befb-79149b3cd0f1-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:31.237904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:30.627379    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c9e5419-faf0-42aa-befb-79149b3cd0f1-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:31.237904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:30.770664    4421 generic.go:356] "Generic (PLEG): container finished" podID="6c9e5419-faf0-42aa-befb-79149b3cd0f1" containerID="94ba640b5d22ebb632c7aa9f78d29859ba6759126c2f7569d33f6d21c67aa315" exitCode=0
Nov 03 20:20:31.237904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:30.770853    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn" event={"ID":"6c9e5419-faf0-42aa-befb-79149b3cd0f1","Type":"ContainerDied","Data":"94ba640b5d22ebb632c7aa9f78d29859ba6759126c2f7569d33f6d21c67aa315"}
Nov 03 20:20:31.237904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:30.770910    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn" event={"ID":"6c9e5419-faf0-42aa-befb-79149b3cd0f1","Type":"ContainerDied","Data":"7fd148ea06b9a3ec96c08f11d1458e7e4ab1e671a000b8b6ad6fd66d30ffac02"}
Nov 03 20:20:31.237904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:30.770942    4421 scope.go:122] "RemoveContainer" containerID="94ba640b5d22ebb632c7aa9f78d29859ba6759126c2f7569d33f6d21c67aa315"
Nov 03 20:20:31.237904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:30.771115    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn"
Nov 03 20:20:31.237904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:30.793570    4421 scope.go:122] "RemoveContainer" containerID="94ba640b5d22ebb632c7aa9f78d29859ba6759126c2f7569d33f6d21c67aa315"
Nov 03 20:20:31.237904 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:20:30.794025    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94ba640b5d22ebb632c7aa9f78d29859ba6759126c2f7569d33f6d21c67aa315\": container with ID starting with 94ba640b5d22ebb632c7aa9f78d29859ba6759126c2f7569d33f6d21c67aa315 not found: ID does not exist" containerID="94ba640b5d22ebb632c7aa9f78d29859ba6759126c2f7569d33f6d21c67aa315"
Nov 03 20:20:31.237904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:30.794056    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94ba640b5d22ebb632c7aa9f78d29859ba6759126c2f7569d33f6d21c67aa315"} err="failed to get container status \"94ba640b5d22ebb632c7aa9f78d29859ba6759126c2f7569d33f6d21c67aa315\": rpc error: code = NotFound desc = could not find container \"94ba640b5d22ebb632c7aa9f78d29859ba6759126c2f7569d33f6d21c67aa315\": container with ID starting with 94ba640b5d22ebb632c7aa9f78d29859ba6759126c2f7569d33f6d21c67aa315 not found: ID does not exist"
Nov 03 20:20:31.237904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:30.803670    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn"]
Nov 03 20:20:31.237904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:30.806273    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9a9ea9-65bdf57f87-s6xvn"]
Nov 03 20:20:31.590677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.590120    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"]
Nov 03 20:20:31.593278 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.591430    4421 cpu_manager.go:414] "containerMap: removing container" podUID="6c9e5419-faf0-42aa-befb-79149b3cd0f1" containerName="maas-api"
Nov 03 20:20:31.593278 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.591457    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c9e5419-faf0-42aa-befb-79149b3cd0f1" containerName="maas-api"
Nov 03 20:20:31.593278 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.591685    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c9e5419-faf0-42aa-befb-79149b3cd0f1" containerName="maas-api"
Nov 03 20:20:31.596888 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.596806    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.610864 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.608761    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f2218a-openshift-default-dockercfg-dtg5j\""
Nov 03 20:20:31.613107 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.612542    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"]
Nov 03 20:20:31.773251 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.773152    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-data\") pod \"e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") " pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.786840 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.786807    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pbbq\" (UniqueName: \"kubernetes.io/projected/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-kube-api-access-9pbbq\") pod \"e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") " pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.787290 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.787176    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-envoy\") pod \"e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") " pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.852626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.851978    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istiod-ca-cert\") pod \"e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") " pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.852626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.852078    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-token\") pod \"e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") " pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.852626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.852323    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-podinfo\") pod \"e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") " pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.852626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.852397    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-workload-socket\") pod \"e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") " pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.852626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.852511    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-credential-socket\") pod \"e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") " pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.852626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.852569    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-workload-certs\") pod \"e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") " pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.960322 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.959998    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-credential-socket\") pod \"e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") " pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.960577 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.960558    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-workload-certs\") pod \"e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") " pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.960801 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.960708    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-data\") pod \"e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") " pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.960801 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.960769    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pbbq\" (UniqueName: \"kubernetes.io/projected/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-kube-api-access-9pbbq\") pod \"e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") " pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.960948 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.960808    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-envoy\") pod \"e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") " pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.960948 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.960859    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-workload-certs\") pod \"e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") " pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.961060 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.960933    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istiod-ca-cert\") pod \"e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") " pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.961060 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.961005    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-token\") pod \"e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") " pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.968228 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.961182    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-podinfo\") pod \"e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") " pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.968228 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.961494    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-data\") pod \"e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") " pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.968228 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.964762    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-workload-socket\") pod \"e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") " pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.968228 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.962941    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istiod-ca-cert\") pod \"e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") " pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.968228 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.964414    4421 generic.go:356] "Generic (PLEG): container finished" podID="249f4a98-95d2-4a3c-9cfc-a6f00e214815" containerID="54c278f026ce64e088b0760f089faf490b21f1ae15859d5a59884713aea1e772" exitCode=0
Nov 03 20:20:31.968228 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.960374    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-credential-socket\") pod \"e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") " pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.968228 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.965271    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-workload-socket\") pod \"e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") " pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.968969 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.964435    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn" event={"ID":"249f4a98-95d2-4a3c-9cfc-a6f00e214815","Type":"ContainerDied","Data":"54c278f026ce64e088b0760f089faf490b21f1ae15859d5a59884713aea1e772"}
Nov 03 20:20:31.969529 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.969512    4421 scope.go:122] "RemoveContainer" containerID="54c278f026ce64e088b0760f089faf490b21f1ae15859d5a59884713aea1e772"
Nov 03 20:20:31.974343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.972663    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:20:31.974343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.972751    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:20:31.974343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.972816    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:20:31.980365 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.980343    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-token\") pod \"e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") " pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.982209 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.982163    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pbbq\" (UniqueName: \"kubernetes.io/projected/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-kube-api-access-9pbbq\") pod \"e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") " pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.984752 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.984686    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-podinfo\") pod \"e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") " pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.992338 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.992313    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-envoy\") pod \"e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") " pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:31.995065 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.993824    4421 generic.go:356] "Generic (PLEG): container finished" podID="9ffda400-707f-48d5-9e1f-7f76caf6cd58" containerID="d488539e315ffe818f2c07db4c61c67d6a4b3d20960e949dc1949477aae66ca8" exitCode=0
Nov 03 20:20:31.995065 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.994018    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh" event={"ID":"9ffda400-707f-48d5-9e1f-7f76caf6cd58","Type":"ContainerDied","Data":"d488539e315ffe818f2c07db4c61c67d6a4b3d20960e949dc1949477aae66ca8"}
Nov 03 20:20:31.995065 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.994472    4421 scope.go:122] "RemoveContainer" containerID="d488539e315ffe818f2c07db4c61c67d6a4b3d20960e949dc1949477aae66ca8"
Nov 03 20:20:31.999536 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.999501    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:20:31.999652 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.999631    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:20:31.999719 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:31.999702    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:20:32.002946 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:32.002920    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg" event={"ID":"f9259c57-03ab-490b-bbde-f40be00f9ad4","Type":"ContainerStarted","Data":"e529299b72cd908a5c4c6432521b49c59460c49640c37d0d281d3ef23595caf4"}
Nov 03 20:20:32.003419 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:32.003398    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg"
Nov 03 20:20:32.005669 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:32.005647    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj" event={"ID":"3ab7a061-3614-41d6-af5a-25dde57e6bbd","Type":"ContainerStarted","Data":"81676fba2a7e805e56718cf592a18c6037d76f11105b5353a157931c9ee73c23"}
Nov 03 20:20:32.005861 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:32.005838    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj"
Nov 03 20:20:32.109773 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:32.109712    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg" podStartSLOduration=2.455961104 podStartE2EDuration="5.109694425s" totalImagesPullingTime="2.653733321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:32.054424832 +0000 UTC m=+1162.187553292" watchObservedRunningTime="2026-11-03 20:20:32.109694425 +0000 UTC m=+1162.242822889"
Nov 03 20:20:32.117785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:32.117725    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj" podStartSLOduration=2.501342642 podStartE2EDuration="5.117707649s" totalImagesPullingTime="2.616365007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:32.115811127 +0000 UTC m=+1162.248939606" watchObservedRunningTime="2026-11-03 20:20:32.117707649 +0000 UTC m=+1162.250836119"
Nov 03 20:20:32.142675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:32.142646    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c9e5419-faf0-42aa-befb-79149b3cd0f1" path="/var/lib/kubelet/pods/6c9e5419-faf0-42aa-befb-79149b3cd0f1/volumes"
Nov 03 20:20:32.237916 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:32.237872    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:32.673929 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:32.673895    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"]
Nov 03 20:20:32.687568 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:32.687524    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:20:32.687717 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:32.687630    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:20:32.687717 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:32.687670    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:20:32.753748 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:32.753715    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:20:33.025583 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:33.025491    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7" event={"ID":"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41","Type":"ContainerStarted","Data":"84c0fd0ca99d70a0ffe4ce66f9ba2d6bb05f0621b8d0e39cb980cf79fd521cc7"}
Nov 03 20:20:33.025875 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:33.025854    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7" event={"ID":"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41","Type":"ContainerStarted","Data":"6e3967d52ffbc3366b3c2a50bd127d6213d934511b611eae40e5ee5d12131590"}
Nov 03 20:20:33.029123 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:33.028721    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn" event={"ID":"249f4a98-95d2-4a3c-9cfc-a6f00e214815","Type":"ContainerStarted","Data":"c81421f21771c03879aabdfdc40c47e1bd18fd88744c2c3d822738cd0ace0e82"}
Nov 03 20:20:33.034799 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:33.034774    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh" event={"ID":"9ffda400-707f-48d5-9e1f-7f76caf6cd58","Type":"ContainerStarted","Data":"90e158f7360be80806deefad728af0023158e6310164b18ffd144e8c090481c7"}
Nov 03 20:20:33.075562 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:33.075503    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7" podStartSLOduration=2.075481321 podStartE2EDuration="2.075481321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:33.072957398 +0000 UTC m=+1163.206085875" watchObservedRunningTime="2026-11-03 20:20:33.075481321 +0000 UTC m=+1163.208609784"
Nov 03 20:20:33.248145 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:33.248115    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:33.248519 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:33.248493    4421 patch_prober.go:29] interesting pod/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 20:20:33.248997 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:33.248721    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7" podUID="6ef60fe1-39dc-4f56-9114-d7d4e79a6b41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 20:20:33.278580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:33.278501    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:20:33.283707 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:33.282707    4421 patch_prober.go:29] interesting pod/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 03 20:20:33.283707 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:33.282807    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn" podUID="249f4a98-95d2-4a3c-9cfc-a6f00e214815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 03 20:20:33.886818 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:33.886776    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"]
Nov 03 20:20:33.891349 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:33.891313    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln" podUID="b5488adb-f734-4d16-81bf-f230fca1306f" containerName="istio-proxy" containerID="cri-o://4f64eed6da03a19d3a70975c168718c97c54c23aeefa8a6f64fd840dc7218325" gracePeriod=30
Nov 03 20:20:33.940453 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:33.940427    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:20:33.940825 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:33.940802    4421 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 03 20:20:33.940990 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:33.940968    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh" podUID="9ffda400-707f-48d5-9e1f-7f76caf6cd58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 03 20:20:34.256785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:34.256736    4421 patch_prober.go:29] interesting pod/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 03 20:20:34.280766 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:34.280708    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn" podUID="249f4a98-95d2-4a3c-9cfc-a6f00e214815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 03 20:20:34.401629 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:34.395551    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:34.939905 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:34.939870    4421 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 03 20:20:34.940322 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:34.939919    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh" podUID="9ffda400-707f-48d5-9e1f-7f76caf6cd58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 03 20:20:35.206772 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:35.206689    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:35.216561 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:35.216528    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:20:35.271850 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:35.271409    4421 patch_prober.go:29] interesting pod/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 03 20:20:35.271850 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:35.271465    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn" podUID="249f4a98-95d2-4a3c-9cfc-a6f00e214815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 03 20:20:35.940344 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:35.940310    4421 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 03 20:20:35.944722 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:35.944682    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh" podUID="9ffda400-707f-48d5-9e1f-7f76caf6cd58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 03 20:20:36.267090 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:36.266998    4421 patch_prober.go:29] interesting pod/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 03 20:20:36.267345 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:36.267315    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn" podUID="249f4a98-95d2-4a3c-9cfc-a6f00e214815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 03 20:20:36.815250 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:36.815218    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82"]
Nov 03 20:20:36.821052 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:36.821024    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82"
Nov 03 20:20:36.821258 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:36.821232    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82"]
Nov 03 20:20:36.871967 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:36.871933    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f2218a\""
Nov 03 20:20:36.887941 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:36.887911    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f2218a-dockercfg-wsc98\""
Nov 03 20:20:36.944901 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:36.944866    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f698c3-76a3-41f3-b9d0-af1ea58a170e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82\" (UID: \"f0f698c3-76a3-41f3-b9d0-af1ea58a170e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82"
Nov 03 20:20:36.945546 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:36.945522    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcd82\" (UniqueName: \"kubernetes.io/projected/f0f698c3-76a3-41f3-b9d0-af1ea58a170e-kube-api-access-gcd82\") pod \"payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82\" (UID: \"f0f698c3-76a3-41f3-b9d0-af1ea58a170e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82"
Nov 03 20:20:36.958715 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:36.952148    4421 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 03 20:20:36.958715 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:36.952198    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh" podUID="9ffda400-707f-48d5-9e1f-7f76caf6cd58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 03 20:20:37.109123 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:37.108998    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f698c3-76a3-41f3-b9d0-af1ea58a170e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82\" (UID: \"f0f698c3-76a3-41f3-b9d0-af1ea58a170e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82"
Nov 03 20:20:37.134685 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:37.120619    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f698c3-76a3-41f3-b9d0-af1ea58a170e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82\" (UID: \"f0f698c3-76a3-41f3-b9d0-af1ea58a170e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82"
Nov 03 20:20:37.146634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:37.135803    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcd82\" (UniqueName: \"kubernetes.io/projected/f0f698c3-76a3-41f3-b9d0-af1ea58a170e-kube-api-access-gcd82\") pod \"payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82\" (UID: \"f0f698c3-76a3-41f3-b9d0-af1ea58a170e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82"
Nov 03 20:20:37.254666 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:37.252407    4421 patch_prober.go:29] interesting pod/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 03 20:20:37.254666 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:37.252572    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn" podUID="249f4a98-95d2-4a3c-9cfc-a6f00e214815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 03 20:20:37.333537 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:37.333487    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcd82\" (UniqueName: \"kubernetes.io/projected/f0f698c3-76a3-41f3-b9d0-af1ea58a170e-kube-api-access-gcd82\") pod \"payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82\" (UID: \"f0f698c3-76a3-41f3-b9d0-af1ea58a170e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82"
Nov 03 20:20:37.438078 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:37.438048    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82"
Nov 03 20:20:37.934142 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:37.934036    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82"]
Nov 03 20:20:37.950646 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:37.944831    4421 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 03 20:20:37.950646 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:37.944906    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh" podUID="9ffda400-707f-48d5-9e1f-7f76caf6cd58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 03 20:20:37.994625 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:20:37.994568    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f698c3_76a3_41f3_b9d0_af1ea58a170e.slice/crio-d8d0dbf3f066a683952b7a95ea563e10dae411e8c41156a279a594d1b00c7b3b WatchSource:0}: Error finding container d8d0dbf3f066a683952b7a95ea563e10dae411e8c41156a279a594d1b00c7b3b: Status 404 returned error can't find the container with id d8d0dbf3f066a683952b7a95ea563e10dae411e8c41156a279a594d1b00c7b3b
Nov 03 20:20:38.272623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:38.271891    4421 patch_prober.go:29] interesting pod/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 03 20:20:38.272623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:38.271950    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn" podUID="249f4a98-95d2-4a3c-9cfc-a6f00e214815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 03 20:20:38.320693 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:38.320654    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82" event={"ID":"f0f698c3-76a3-41f3-b9d0-af1ea58a170e","Type":"ContainerStarted","Data":"8ac6e95e328c23763fcc4612dbd8c4688ba56cbbc6c597afff86f2c68b611f85"}
Nov 03 20:20:38.320955 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:38.320933    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82" event={"ID":"f0f698c3-76a3-41f3-b9d0-af1ea58a170e","Type":"ContainerStarted","Data":"d8d0dbf3f066a683952b7a95ea563e10dae411e8c41156a279a594d1b00c7b3b"}
Nov 03 20:20:38.321096 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:38.321084    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82"
Nov 03 20:20:38.344623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:38.341852    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82" podStartSLOduration=2.341835802 podStartE2EDuration="2.341835802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:38.339337691 +0000 UTC m=+1168.472466165" watchObservedRunningTime="2026-11-03 20:20:38.341835802 +0000 UTC m=+1168.474964267"
Nov 03 20:20:38.962891 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:38.959748    4421 patch_prober.go:29] interesting pod/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 03 20:20:38.962891 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:38.959819    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh" podUID="9ffda400-707f-48d5-9e1f-7f76caf6cd58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 03 20:20:39.128057 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:20:39.128024    4421 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5488adb_f734_4d16_81bf_f230fca1306f.slice/crio-4f64eed6da03a19d3a70975c168718c97c54c23aeefa8a6f64fd840dc7218325.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:20:39.252214 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.252128    4421 patch_prober.go:29] interesting pod/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Nov 03 20:20:39.252214 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.252196    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn" podUID="249f4a98-95d2-4a3c-9cfc-a6f00e214815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Nov 03 20:20:39.295384 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.295359    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:20:39.380480 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.380427    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5488adb-f734-4d16-81bf-f230fca1306f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5488adb-f734-4d16-81bf-f230fca1306f-istio-podinfo\") pod \"b5488adb-f734-4d16-81bf-f230fca1306f\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") "
Nov 03 20:20:39.380724 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.380587    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-istio-envoy\") pod \"b5488adb-f734-4d16-81bf-f230fca1306f\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") "
Nov 03 20:20:39.380724 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.380717    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-workload-socket\") pod \"b5488adb-f734-4d16-81bf-f230fca1306f\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") "
Nov 03 20:20:39.380889 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.380770    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-istio-data\") pod \"b5488adb-f734-4d16-81bf-f230fca1306f\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") "
Nov 03 20:20:39.380889 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.380806    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-credential-socket\") pod \"b5488adb-f734-4d16-81bf-f230fca1306f\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") "
Nov 03 20:20:39.380889 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.380863    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5488adb-f734-4d16-81bf-f230fca1306f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5488adb-f734-4d16-81bf-f230fca1306f-istiod-ca-cert\") pod \"b5488adb-f734-4d16-81bf-f230fca1306f\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") "
Nov 03 20:20:39.381056 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.380927    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5488adb-f734-4d16-81bf-f230fca1306f-kube-api-access-dzq7x\" (UniqueName: \"kubernetes.io/projected/b5488adb-f734-4d16-81bf-f230fca1306f-kube-api-access-dzq7x\") pod \"b5488adb-f734-4d16-81bf-f230fca1306f\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") "
Nov 03 20:20:39.381056 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.380986    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5488adb-f734-4d16-81bf-f230fca1306f-istio-token\" (UniqueName: \"kubernetes.io/projected/b5488adb-f734-4d16-81bf-f230fca1306f-istio-token\") pod \"b5488adb-f734-4d16-81bf-f230fca1306f\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") "
Nov 03 20:20:39.381165 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.381136    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-workload-certs\") pod \"b5488adb-f734-4d16-81bf-f230fca1306f\" (UID: \"b5488adb-f734-4d16-81bf-f230fca1306f\") "
Nov 03 20:20:39.383198 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.383163    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-credential-socket" pod "b5488adb-f734-4d16-81bf-f230fca1306f" (UID: "b5488adb-f734-4d16-81bf-f230fca1306f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:39.383627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.383580    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-istio-data" pod "b5488adb-f734-4d16-81bf-f230fca1306f" (UID: "b5488adb-f734-4d16-81bf-f230fca1306f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:39.383799 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.383693    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-workload-certs" pod "b5488adb-f734-4d16-81bf-f230fca1306f" (UID: "b5488adb-f734-4d16-81bf-f230fca1306f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:39.385442 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.385417    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5488adb-f734-4d16-81bf-f230fca1306f-istiod-ca-cert" pod "b5488adb-f734-4d16-81bf-f230fca1306f" (UID: "b5488adb-f734-4d16-81bf-f230fca1306f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:39.385546 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.385495    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-workload-socket" pod "b5488adb-f734-4d16-81bf-f230fca1306f" (UID: "b5488adb-f734-4d16-81bf-f230fca1306f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:39.387958 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.387336    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-istio-envoy" pod "b5488adb-f734-4d16-81bf-f230fca1306f" (UID: "b5488adb-f734-4d16-81bf-f230fca1306f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:20:39.388973 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.388943    4421 generic.go:356] "Generic (PLEG): container finished" podID="b5488adb-f734-4d16-81bf-f230fca1306f" containerID="4f64eed6da03a19d3a70975c168718c97c54c23aeefa8a6f64fd840dc7218325" exitCode=0
Nov 03 20:20:39.389323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.389062    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln" event={"ID":"b5488adb-f734-4d16-81bf-f230fca1306f","Type":"ContainerDied","Data":"4f64eed6da03a19d3a70975c168718c97c54c23aeefa8a6f64fd840dc7218325"}
Nov 03 20:20:39.389323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.389154    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln" event={"ID":"b5488adb-f734-4d16-81bf-f230fca1306f","Type":"ContainerDied","Data":"2d07ac361ba739400938b903b3e29aaf1aba98b8353e0d00a89fa0f96ac5e082"}
Nov 03 20:20:39.389323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.389187    4421 scope.go:122] "RemoveContainer" containerID="4f64eed6da03a19d3a70975c168718c97c54c23aeefa8a6f64fd840dc7218325"
Nov 03 20:20:39.389323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.389040    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"
Nov 03 20:20:39.389323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.389195    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5488adb-f734-4d16-81bf-f230fca1306f-istio-podinfo" pod "b5488adb-f734-4d16-81bf-f230fca1306f" (UID: "b5488adb-f734-4d16-81bf-f230fca1306f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:20:39.391034 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.390978    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5488adb-f734-4d16-81bf-f230fca1306f-kube-api-access-dzq7x" pod "b5488adb-f734-4d16-81bf-f230fca1306f" (UID: "b5488adb-f734-4d16-81bf-f230fca1306f"). InnerVolumeSpecName "kube-api-access-dzq7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:39.391228 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.391207    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5488adb-f734-4d16-81bf-f230fca1306f-istio-token" pod "b5488adb-f734-4d16-81bf-f230fca1306f" (UID: "b5488adb-f734-4d16-81bf-f230fca1306f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:39.432259 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.432232    4421 scope.go:122] "RemoveContainer" containerID="bd20d3285ea2dbd64059ed5b469e184320bfb51da263079d47d02b26fd6b69bd"
Nov 03 20:20:39.443117 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.443075    4421 scope.go:122] "RemoveContainer" containerID="4f64eed6da03a19d3a70975c168718c97c54c23aeefa8a6f64fd840dc7218325"
Nov 03 20:20:39.443478 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:20:39.443455    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f64eed6da03a19d3a70975c168718c97c54c23aeefa8a6f64fd840dc7218325\": container with ID starting with 4f64eed6da03a19d3a70975c168718c97c54c23aeefa8a6f64fd840dc7218325 not found: ID does not exist" containerID="4f64eed6da03a19d3a70975c168718c97c54c23aeefa8a6f64fd840dc7218325"
Nov 03 20:20:39.443576 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.443488    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f64eed6da03a19d3a70975c168718c97c54c23aeefa8a6f64fd840dc7218325"} err="failed to get container status \"4f64eed6da03a19d3a70975c168718c97c54c23aeefa8a6f64fd840dc7218325\": rpc error: code = NotFound desc = could not find container \"4f64eed6da03a19d3a70975c168718c97c54c23aeefa8a6f64fd840dc7218325\": container with ID starting with 4f64eed6da03a19d3a70975c168718c97c54c23aeefa8a6f64fd840dc7218325 not found: ID does not exist"
Nov 03 20:20:39.443576 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.443506    4421 scope.go:122] "RemoveContainer" containerID="bd20d3285ea2dbd64059ed5b469e184320bfb51da263079d47d02b26fd6b69bd"
Nov 03 20:20:39.443846 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:20:39.443822    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd20d3285ea2dbd64059ed5b469e184320bfb51da263079d47d02b26fd6b69bd\": container with ID starting with bd20d3285ea2dbd64059ed5b469e184320bfb51da263079d47d02b26fd6b69bd not found: ID does not exist" containerID="bd20d3285ea2dbd64059ed5b469e184320bfb51da263079d47d02b26fd6b69bd"
Nov 03 20:20:39.443911 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.443857    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd20d3285ea2dbd64059ed5b469e184320bfb51da263079d47d02b26fd6b69bd"} err="failed to get container status \"bd20d3285ea2dbd64059ed5b469e184320bfb51da263079d47d02b26fd6b69bd\": rpc error: code = NotFound desc = could not find container \"bd20d3285ea2dbd64059ed5b469e184320bfb51da263079d47d02b26fd6b69bd\": container with ID starting with bd20d3285ea2dbd64059ed5b469e184320bfb51da263079d47d02b26fd6b69bd not found: ID does not exist"
Nov 03 20:20:39.484240 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.484197    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:39.484240 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.484235    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:39.484445 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.484250    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:39.484445 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.484264    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:39.484445 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.484279    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5488adb-f734-4d16-81bf-f230fca1306f-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:39.484445 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.484292    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzq7x\" (UniqueName: \"kubernetes.io/projected/b5488adb-f734-4d16-81bf-f230fca1306f-kube-api-access-dzq7x\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:39.484445 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.484305    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5488adb-f734-4d16-81bf-f230fca1306f-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:39.484445 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.484317    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5488adb-f734-4d16-81bf-f230fca1306f-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:39.484445 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.484328    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5488adb-f734-4d16-81bf-f230fca1306f-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:39.715517 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.715490    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"]
Nov 03 20:20:39.721222 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.721195    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9a9ea9-openshift-default-b548dcd8f-zj7ln"]
Nov 03 20:20:39.944691 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.944665    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:20:39.945178 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.945160    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:20:39.945956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:39.945935    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:20:40.020067 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:40.019974    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5488adb-f734-4d16-81bf-f230fca1306f" path="/var/lib/kubelet/pods/b5488adb-f734-4d16-81bf-f230fca1306f/volumes"
Nov 03 20:20:40.256797 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:40.256756    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:20:40.257171 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:40.257142    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:20:40.257957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:40.257935    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:20:40.475853 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:40.475822    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g"]
Nov 03 20:20:40.486269 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:40.486239    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx"]
Nov 03 20:20:41.539328 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.539286    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"]
Nov 03 20:20:41.540343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.540321    4421 cpu_manager.go:414] "containerMap: removing container" podUID="b5488adb-f734-4d16-81bf-f230fca1306f" containerName="istio-proxy"
Nov 03 20:20:41.540438 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.540344    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5488adb-f734-4d16-81bf-f230fca1306f" containerName="istio-proxy"
Nov 03 20:20:41.540583 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.540569    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5488adb-f734-4d16-81bf-f230fca1306f" containerName="istio-proxy"
Nov 03 20:20:41.544028 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.544004    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.554790 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.554047    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b07e26-openshift-default-dockercfg-qvzq4\""
Nov 03 20:20:41.560843 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.560818    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"]
Nov 03 20:20:41.622936 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.622905    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10a12293-4a77-4829-93da-7e1909f3853f-istio-token\") pod \"e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") " pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.623116 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.622944    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10a12293-4a77-4829-93da-7e1909f3853f-istiod-ca-cert\") pod \"e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") " pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.623174 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.623129    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10a12293-4a77-4829-93da-7e1909f3853f-istio-podinfo\") pod \"e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") " pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.623239 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.623176    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-istio-data\") pod \"e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") " pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.623239 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.623208    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8wxr\" (UniqueName: \"kubernetes.io/projected/10a12293-4a77-4829-93da-7e1909f3853f-kube-api-access-x8wxr\") pod \"e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") " pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.623341 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.623299    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-workload-socket\") pod \"e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") " pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.623341 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.623328    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-workload-certs\") pod \"e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") " pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.623473 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.623457    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-istio-envoy\") pod \"e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") " pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.623519 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.623495    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-credential-socket\") pod \"e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") " pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.736636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.725271    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10a12293-4a77-4829-93da-7e1909f3853f-istio-podinfo\") pod \"e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") " pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.736636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.725345    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-istio-data\") pod \"e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") " pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.736636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.725391    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8wxr\" (UniqueName: \"kubernetes.io/projected/10a12293-4a77-4829-93da-7e1909f3853f-kube-api-access-x8wxr\") pod \"e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") " pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.736636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.725452    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-workload-socket\") pod \"e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") " pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.736636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.725483    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-workload-certs\") pod \"e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") " pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.744208 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.744175    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-istio-envoy\") pod \"e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") " pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.766768 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.766730    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-credential-socket\") pod \"e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") " pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.767199 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.766935    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-workload-socket\") pod \"e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") " pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.767199 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.767170    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-workload-certs\") pod \"e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") " pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.767199 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.767176    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10a12293-4a77-4829-93da-7e1909f3853f-istio-token\") pod \"e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") " pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.767425 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.767250    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10a12293-4a77-4829-93da-7e1909f3853f-istiod-ca-cert\") pod \"e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") " pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.773765 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.768136    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10a12293-4a77-4829-93da-7e1909f3853f-istiod-ca-cert\") pod \"e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") " pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.789063 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.788977    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-istio-envoy\") pod \"e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") " pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.799581 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.799455    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8wxr\" (UniqueName: \"kubernetes.io/projected/10a12293-4a77-4829-93da-7e1909f3853f-kube-api-access-x8wxr\") pod \"e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") " pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.809634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.801712    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10a12293-4a77-4829-93da-7e1909f3853f-istio-podinfo\") pod \"e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") " pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.809634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.804939    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10a12293-4a77-4829-93da-7e1909f3853f-istio-token\") pod \"e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") " pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.816892 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.813911    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-istio-data\") pod \"e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") " pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.816892 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.816846    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-credential-socket\") pod \"e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") " pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:41.864163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:41.863884    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:42.284568 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:42.284532    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"]
Nov 03 20:20:42.295804 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:42.295753    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:20:42.295921 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:42.295899    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:20:42.296002 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:42.295980    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:20:42.319419 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:42.319394    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr"]
Nov 03 20:20:42.319776 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:42.319726    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr" podUID="0280365a-1ebd-4513-8cc6-ec821dc04960" containerName="maas-api" containerID="cri-o://272bb0d1ba70ce77de801d80bbeaac9a0436f74b241f5908634ef4bc6eb673c4" gracePeriod=30
Nov 03 20:20:42.415147 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:42.415113    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh" event={"ID":"10a12293-4a77-4829-93da-7e1909f3853f","Type":"ContainerStarted","Data":"ed1984d7ee9aaf93e1844b248c934a8471e250d05b14b1866df5474efe57cb54"}
Nov 03 20:20:42.927771 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:42.927070    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr"
Nov 03 20:20:43.037202 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:43.037166    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg"
Nov 03 20:20:43.037393 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:43.037237    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj"
Nov 03 20:20:43.042347 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:43.042316    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0280365a-1ebd-4513-8cc6-ec821dc04960-kube-api-access-wvhbv\" (UniqueName: \"kubernetes.io/projected/0280365a-1ebd-4513-8cc6-ec821dc04960-kube-api-access-wvhbv\") pod \"0280365a-1ebd-4513-8cc6-ec821dc04960\" (UID: \"0280365a-1ebd-4513-8cc6-ec821dc04960\") "
Nov 03 20:20:43.042347 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:43.042351    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0280365a-1ebd-4513-8cc6-ec821dc04960-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0280365a-1ebd-4513-8cc6-ec821dc04960-metrics-certs\") pod \"0280365a-1ebd-4513-8cc6-ec821dc04960\" (UID: \"0280365a-1ebd-4513-8cc6-ec821dc04960\") "
Nov 03 20:20:43.042576 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:43.042559    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0280365a-1ebd-4513-8cc6-ec821dc04960-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0280365a-1ebd-4513-8cc6-ec821dc04960-maas-api-tls\") pod \"0280365a-1ebd-4513-8cc6-ec821dc04960\" (UID: \"0280365a-1ebd-4513-8cc6-ec821dc04960\") "
Nov 03 20:20:43.042679 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:43.042650    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0280365a-1ebd-4513-8cc6-ec821dc04960-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0280365a-1ebd-4513-8cc6-ec821dc04960-openshift-service-ca\") pod \"0280365a-1ebd-4513-8cc6-ec821dc04960\" (UID: \"0280365a-1ebd-4513-8cc6-ec821dc04960\") "
Nov 03 20:20:43.043031 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:43.043003    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0280365a-1ebd-4513-8cc6-ec821dc04960-openshift-service-ca" pod "0280365a-1ebd-4513-8cc6-ec821dc04960" (UID: "0280365a-1ebd-4513-8cc6-ec821dc04960"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:43.043374 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:43.043353    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0280365a-1ebd-4513-8cc6-ec821dc04960-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:43.045078 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:43.045045    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0280365a-1ebd-4513-8cc6-ec821dc04960-kube-api-access-wvhbv" pod "0280365a-1ebd-4513-8cc6-ec821dc04960" (UID: "0280365a-1ebd-4513-8cc6-ec821dc04960"). InnerVolumeSpecName "kube-api-access-wvhbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:43.045181 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:43.045134    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0280365a-1ebd-4513-8cc6-ec821dc04960-metrics-certs" pod "0280365a-1ebd-4513-8cc6-ec821dc04960" (UID: "0280365a-1ebd-4513-8cc6-ec821dc04960"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:43.045279 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:43.045263    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0280365a-1ebd-4513-8cc6-ec821dc04960-maas-api-tls" pod "0280365a-1ebd-4513-8cc6-ec821dc04960" (UID: "0280365a-1ebd-4513-8cc6-ec821dc04960"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:20:43.144933 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:43.144858    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0280365a-1ebd-4513-8cc6-ec821dc04960-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:43.144933 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:43.144891    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvhbv\" (UniqueName: \"kubernetes.io/projected/0280365a-1ebd-4513-8cc6-ec821dc04960-kube-api-access-wvhbv\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:43.144933 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:43.144906    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0280365a-1ebd-4513-8cc6-ec821dc04960-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:43.425824 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:43.425792    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh" event={"ID":"10a12293-4a77-4829-93da-7e1909f3853f","Type":"ContainerStarted","Data":"c37b2c431f861865fb9b00217e1225156c455c23ff225fa7331b785bb7cbcaf1"}
Nov 03 20:20:43.427251 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:43.427224    4421 generic.go:356] "Generic (PLEG): container finished" podID="0280365a-1ebd-4513-8cc6-ec821dc04960" containerID="272bb0d1ba70ce77de801d80bbeaac9a0436f74b241f5908634ef4bc6eb673c4" exitCode=0
Nov 03 20:20:43.427385 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:43.427259    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr"
Nov 03 20:20:43.427385 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:43.427299    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr" event={"ID":"0280365a-1ebd-4513-8cc6-ec821dc04960","Type":"ContainerDied","Data":"272bb0d1ba70ce77de801d80bbeaac9a0436f74b241f5908634ef4bc6eb673c4"}
Nov 03 20:20:43.427385 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:43.427343    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr" event={"ID":"0280365a-1ebd-4513-8cc6-ec821dc04960","Type":"ContainerDied","Data":"195f55df64ee23689dde2018cb737bce6ec92dadd3604518d417618e78c002df"}
Nov 03 20:20:43.427385 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:43.427365    4421 scope.go:122] "RemoveContainer" containerID="272bb0d1ba70ce77de801d80bbeaac9a0436f74b241f5908634ef4bc6eb673c4"
Nov 03 20:20:43.439027 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:43.439012    4421 scope.go:122] "RemoveContainer" containerID="272bb0d1ba70ce77de801d80bbeaac9a0436f74b241f5908634ef4bc6eb673c4"
Nov 03 20:20:43.439315 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:20:43.439295    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"272bb0d1ba70ce77de801d80bbeaac9a0436f74b241f5908634ef4bc6eb673c4\": container with ID starting with 272bb0d1ba70ce77de801d80bbeaac9a0436f74b241f5908634ef4bc6eb673c4 not found: ID does not exist" containerID="272bb0d1ba70ce77de801d80bbeaac9a0436f74b241f5908634ef4bc6eb673c4"
Nov 03 20:20:43.439390 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:43.439321    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"272bb0d1ba70ce77de801d80bbeaac9a0436f74b241f5908634ef4bc6eb673c4"} err="failed to get container status \"272bb0d1ba70ce77de801d80bbeaac9a0436f74b241f5908634ef4bc6eb673c4\": rpc error: code = NotFound desc = could not find container \"272bb0d1ba70ce77de801d80bbeaac9a0436f74b241f5908634ef4bc6eb673c4\": container with ID starting with 272bb0d1ba70ce77de801d80bbeaac9a0436f74b241f5908634ef4bc6eb673c4 not found: ID does not exist"
Nov 03 20:20:43.449437 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:43.449395    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh" podStartSLOduration=2.449382013 podStartE2EDuration="2.449382013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:43.448456993 +0000 UTC m=+1173.581585466" watchObservedRunningTime="2026-11-03 20:20:43.449382013 +0000 UTC m=+1173.582510506"
Nov 03 20:20:43.463982 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:43.463959    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr"]
Nov 03 20:20:43.469579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:43.469559    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1127f5-776757c898-j42gr"]
Nov 03 20:20:43.865194 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:43.865152    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:43.871035 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:43.871010    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:44.017500 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:44.017469    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0280365a-1ebd-4513-8cc6-ec821dc04960" path="/var/lib/kubelet/pods/0280365a-1ebd-4513-8cc6-ec821dc04960/volumes"
Nov 03 20:20:44.417864 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:44.417830    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85"]
Nov 03 20:20:44.418585 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:44.418554    4421 cpu_manager.go:414] "containerMap: removing container" podUID="0280365a-1ebd-4513-8cc6-ec821dc04960" containerName="maas-api"
Nov 03 20:20:44.418585 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:44.418575    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0280365a-1ebd-4513-8cc6-ec821dc04960" containerName="maas-api"
Nov 03 20:20:44.418782 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:44.418667    4421 cpu_manager.go:414] "containerMap: removing container" podUID="b5488adb-f734-4d16-81bf-f230fca1306f" containerName="istio-proxy"
Nov 03 20:20:44.418782 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:44.418687    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5488adb-f734-4d16-81bf-f230fca1306f" containerName="istio-proxy"
Nov 03 20:20:44.418898 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:44.418867    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5488adb-f734-4d16-81bf-f230fca1306f" containerName="istio-proxy"
Nov 03 20:20:44.418898 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:44.418883    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0280365a-1ebd-4513-8cc6-ec821dc04960" containerName="maas-api"
Nov 03 20:20:44.424096 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:44.424071    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85"
Nov 03 20:20:44.433218 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:44.433195    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f2218a-serving-cert\""
Nov 03 20:20:44.447361 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:44.447312    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85"]
Nov 03 20:20:44.449333 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:44.449310    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:44.450394 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:44.450373    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:20:44.585868 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:44.585834    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-openshift-service-ca\") pod \"maas-api-e2e-api-a-f2218a-867759db75-qgh85\" (UID: \"5906e3ff-fc91-4434-93b1-3cd8c77dbae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85"
Nov 03 20:20:44.586056 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:44.585881    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-metrics-certs\") pod \"maas-api-e2e-api-a-f2218a-867759db75-qgh85\" (UID: \"5906e3ff-fc91-4434-93b1-3cd8c77dbae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85"
Nov 03 20:20:44.586204 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:44.586179    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcp75\" (UniqueName: \"kubernetes.io/projected/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-kube-api-access-bcp75\") pod \"maas-api-e2e-api-a-f2218a-867759db75-qgh85\" (UID: \"5906e3ff-fc91-4434-93b1-3cd8c77dbae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85"
Nov 03 20:20:44.586427 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:44.586410    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-maas-api-tls\") pod \"maas-api-e2e-api-a-f2218a-867759db75-qgh85\" (UID: \"5906e3ff-fc91-4434-93b1-3cd8c77dbae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85"
Nov 03 20:20:44.690965 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:44.690929    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcp75\" (UniqueName: \"kubernetes.io/projected/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-kube-api-access-bcp75\") pod \"maas-api-e2e-api-a-f2218a-867759db75-qgh85\" (UID: \"5906e3ff-fc91-4434-93b1-3cd8c77dbae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85"
Nov 03 20:20:44.697481 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:44.691735    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-maas-api-tls\") pod \"maas-api-e2e-api-a-f2218a-867759db75-qgh85\" (UID: \"5906e3ff-fc91-4434-93b1-3cd8c77dbae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85"
Nov 03 20:20:44.698576 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:44.698547    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-openshift-service-ca\") pod \"maas-api-e2e-api-a-f2218a-867759db75-qgh85\" (UID: \"5906e3ff-fc91-4434-93b1-3cd8c77dbae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85"
Nov 03 20:20:44.698846 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:44.698825    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-metrics-certs\") pod \"maas-api-e2e-api-a-f2218a-867759db75-qgh85\" (UID: \"5906e3ff-fc91-4434-93b1-3cd8c77dbae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85"
Nov 03 20:20:44.707702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:44.699590    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-openshift-service-ca\") pod \"maas-api-e2e-api-a-f2218a-867759db75-qgh85\" (UID: \"5906e3ff-fc91-4434-93b1-3cd8c77dbae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85"
Nov 03 20:20:44.710502 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:44.710470    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-maas-api-tls\") pod \"maas-api-e2e-api-a-f2218a-867759db75-qgh85\" (UID: \"5906e3ff-fc91-4434-93b1-3cd8c77dbae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85"
Nov 03 20:20:44.714306 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:44.714272    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcp75\" (UniqueName: \"kubernetes.io/projected/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-kube-api-access-bcp75\") pod \"maas-api-e2e-api-a-f2218a-867759db75-qgh85\" (UID: \"5906e3ff-fc91-4434-93b1-3cd8c77dbae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85"
Nov 03 20:20:44.715967 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:44.714736    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-metrics-certs\") pod \"maas-api-e2e-api-a-f2218a-867759db75-qgh85\" (UID: \"5906e3ff-fc91-4434-93b1-3cd8c77dbae3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85"
Nov 03 20:20:44.763239 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:44.763207    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85"
Nov 03 20:20:45.052153 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:45.052128    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85"]
Nov 03 20:20:45.052577 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:20:45.052337    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5906e3ff_fc91_4434_93b1_3cd8c77dbae3.slice/crio-7ac4334f6f15a5321f9665d9204c16c62a8c11b7f7161f30f9b4d01805b3441d WatchSource:0}: Error finding container 7ac4334f6f15a5321f9665d9204c16c62a8c11b7f7161f30f9b4d01805b3441d: Status 404 returned error can't find the container with id 7ac4334f6f15a5321f9665d9204c16c62a8c11b7f7161f30f9b4d01805b3441d
Nov 03 20:20:45.458197 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:45.458162    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85" event={"ID":"5906e3ff-fc91-4434-93b1-3cd8c77dbae3","Type":"ContainerStarted","Data":"7ac4334f6f15a5321f9665d9204c16c62a8c11b7f7161f30f9b4d01805b3441d"}
Nov 03 20:20:45.476185 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:45.476147    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g" podUID="c8adedde-7209-409b-a289-aec02b4e6add" containerName="payload-processing" containerID="cri-o://8fc0aaf8d9ff4a44c57c62915771f13cf7561ce3d614486e56d758a86348fd6a" gracePeriod=25
Nov 03 20:20:45.487577 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:45.487548    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx" podUID="7f7b664e-2b08-458d-bd6f-324ff24f8441" containerName="payload-pre-processing" containerID="cri-o://a466a3757af60e6e4f9abc678ae234e4c5894b9af7d321088f4e34d4ed6379ef" gracePeriod=25
Nov 03 20:20:45.971112 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:20:45.897293    4421 logging.go:55] [core] [Channel #1184 SubChannel #1185]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.107:50052", ServerName: "10.134.0.107:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.107:50052: connect: connection refused"
Nov 03 20:20:46.109507 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.109472    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g"
Nov 03 20:20:46.181089 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.181054    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx"
Nov 03 20:20:46.232163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.232133    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8adedde-7209-409b-a289-aec02b4e6add-kube-api-access-tkrmj\" (UniqueName: \"kubernetes.io/projected/c8adedde-7209-409b-a289-aec02b4e6add-kube-api-access-tkrmj\") pod \"c8adedde-7209-409b-a289-aec02b4e6add\" (UID: \"c8adedde-7209-409b-a289-aec02b4e6add\") "
Nov 03 20:20:46.232350 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.232316    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8adedde-7209-409b-a289-aec02b4e6add-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8adedde-7209-409b-a289-aec02b4e6add-plugins-config-volume\") pod \"c8adedde-7209-409b-a289-aec02b4e6add\" (UID: \"c8adedde-7209-409b-a289-aec02b4e6add\") "
Nov 03 20:20:46.234559 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.234526    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8adedde-7209-409b-a289-aec02b4e6add-plugins-config-volume" pod "c8adedde-7209-409b-a289-aec02b4e6add" (UID: "c8adedde-7209-409b-a289-aec02b4e6add"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:46.236003 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.235976    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8adedde-7209-409b-a289-aec02b4e6add-kube-api-access-tkrmj" pod "c8adedde-7209-409b-a289-aec02b4e6add" (UID: "c8adedde-7209-409b-a289-aec02b4e6add"). InnerVolumeSpecName "kube-api-access-tkrmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:46.334031 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.333949    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f7b664e-2b08-458d-bd6f-324ff24f8441-kube-api-access-gz5v6\" (UniqueName: \"kubernetes.io/projected/7f7b664e-2b08-458d-bd6f-324ff24f8441-kube-api-access-gz5v6\") pod \"7f7b664e-2b08-458d-bd6f-324ff24f8441\" (UID: \"7f7b664e-2b08-458d-bd6f-324ff24f8441\") "
Nov 03 20:20:46.334031 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.334015    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f7b664e-2b08-458d-bd6f-324ff24f8441-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7b664e-2b08-458d-bd6f-324ff24f8441-plugins-config-volume\") pod \"7f7b664e-2b08-458d-bd6f-324ff24f8441\" (UID: \"7f7b664e-2b08-458d-bd6f-324ff24f8441\") "
Nov 03 20:20:46.334424 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.334401    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f7b664e-2b08-458d-bd6f-324ff24f8441-plugins-config-volume" pod "7f7b664e-2b08-458d-bd6f-324ff24f8441" (UID: "7f7b664e-2b08-458d-bd6f-324ff24f8441"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:46.334580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.334561    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8adedde-7209-409b-a289-aec02b4e6add-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:46.334660 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.334587    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7b664e-2b08-458d-bd6f-324ff24f8441-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:46.334660 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.334618    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkrmj\" (UniqueName: \"kubernetes.io/projected/c8adedde-7209-409b-a289-aec02b4e6add-kube-api-access-tkrmj\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:46.336462 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.336437    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f7b664e-2b08-458d-bd6f-324ff24f8441-kube-api-access-gz5v6" pod "7f7b664e-2b08-458d-bd6f-324ff24f8441" (UID: "7f7b664e-2b08-458d-bd6f-324ff24f8441"). InnerVolumeSpecName "kube-api-access-gz5v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:46.442107 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.440922    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz5v6\" (UniqueName: \"kubernetes.io/projected/7f7b664e-2b08-458d-bd6f-324ff24f8441-kube-api-access-gz5v6\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:46.517635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.514998    4421 generic.go:356] "Generic (PLEG): container finished" podID="c8adedde-7209-409b-a289-aec02b4e6add" containerID="8fc0aaf8d9ff4a44c57c62915771f13cf7561ce3d614486e56d758a86348fd6a" exitCode=0
Nov 03 20:20:46.517635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.515238    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g" event={"ID":"c8adedde-7209-409b-a289-aec02b4e6add","Type":"ContainerDied","Data":"8fc0aaf8d9ff4a44c57c62915771f13cf7561ce3d614486e56d758a86348fd6a"}
Nov 03 20:20:46.517635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.515322    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g" event={"ID":"c8adedde-7209-409b-a289-aec02b4e6add","Type":"ContainerDied","Data":"423c86c771334a6bb89615861d6225feee8c75d7a54f47bacae600c9b2180145"}
Nov 03 20:20:46.517635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.515363    4421 scope.go:122] "RemoveContainer" containerID="8fc0aaf8d9ff4a44c57c62915771f13cf7561ce3d614486e56d758a86348fd6a"
Nov 03 20:20:46.517635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.515512    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g"
Nov 03 20:20:46.528851 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.528790    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85" event={"ID":"5906e3ff-fc91-4434-93b1-3cd8c77dbae3","Type":"ContainerStarted","Data":"05b01b4040b44364f00313b3cdf743ce43be987ba70c69acdb38a9f01abcb32e"}
Nov 03 20:20:46.530248 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.530159    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85"
Nov 03 20:20:46.537991 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.537957    4421 generic.go:356] "Generic (PLEG): container finished" podID="7f7b664e-2b08-458d-bd6f-324ff24f8441" containerID="a466a3757af60e6e4f9abc678ae234e4c5894b9af7d321088f4e34d4ed6379ef" exitCode=0
Nov 03 20:20:46.539235 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.539212    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx"
Nov 03 20:20:46.540157 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.539402    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx" event={"ID":"7f7b664e-2b08-458d-bd6f-324ff24f8441","Type":"ContainerDied","Data":"a466a3757af60e6e4f9abc678ae234e4c5894b9af7d321088f4e34d4ed6379ef"}
Nov 03 20:20:46.540157 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.540055    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx" event={"ID":"7f7b664e-2b08-458d-bd6f-324ff24f8441","Type":"ContainerDied","Data":"1e10234d5b54c1364288b67923b444b79c44f7aa45789ed47ed06e0af01d9057"}
Nov 03 20:20:46.602640 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.598944    4421 scope.go:122] "RemoveContainer" containerID="8fc0aaf8d9ff4a44c57c62915771f13cf7561ce3d614486e56d758a86348fd6a"
Nov 03 20:20:46.602640 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:20:46.601102    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fc0aaf8d9ff4a44c57c62915771f13cf7561ce3d614486e56d758a86348fd6a\": container with ID starting with 8fc0aaf8d9ff4a44c57c62915771f13cf7561ce3d614486e56d758a86348fd6a not found: ID does not exist" containerID="8fc0aaf8d9ff4a44c57c62915771f13cf7561ce3d614486e56d758a86348fd6a"
Nov 03 20:20:46.602640 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.601157    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fc0aaf8d9ff4a44c57c62915771f13cf7561ce3d614486e56d758a86348fd6a"} err="failed to get container status \"8fc0aaf8d9ff4a44c57c62915771f13cf7561ce3d614486e56d758a86348fd6a\": rpc error: code = NotFound desc = could not find container \"8fc0aaf8d9ff4a44c57c62915771f13cf7561ce3d614486e56d758a86348fd6a\": container with ID starting with 8fc0aaf8d9ff4a44c57c62915771f13cf7561ce3d614486e56d758a86348fd6a not found: ID does not exist"
Nov 03 20:20:46.602640 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.601183    4421 scope.go:122] "RemoveContainer" containerID="a466a3757af60e6e4f9abc678ae234e4c5894b9af7d321088f4e34d4ed6379ef"
Nov 03 20:20:46.609095 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.606043    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85" podStartSLOduration=2.115088865 podStartE2EDuration="2.606022394s" totalImagesPullingTime="490.933529ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:46.556883523 +0000 UTC m=+1176.690011985" watchObservedRunningTime="2026-11-03 20:20:46.606022394 +0000 UTC m=+1176.739150857"
Nov 03 20:20:46.609095 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.607108    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g"]
Nov 03 20:20:46.613577 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.613517    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g"]
Nov 03 20:20:46.639554 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.639530    4421 scope.go:122] "RemoveContainer" containerID="a466a3757af60e6e4f9abc678ae234e4c5894b9af7d321088f4e34d4ed6379ef"
Nov 03 20:20:46.640205 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:20:46.640175    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a466a3757af60e6e4f9abc678ae234e4c5894b9af7d321088f4e34d4ed6379ef\": container with ID starting with a466a3757af60e6e4f9abc678ae234e4c5894b9af7d321088f4e34d4ed6379ef not found: ID does not exist" containerID="a466a3757af60e6e4f9abc678ae234e4c5894b9af7d321088f4e34d4ed6379ef"
Nov 03 20:20:46.640388 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.640355    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a466a3757af60e6e4f9abc678ae234e4c5894b9af7d321088f4e34d4ed6379ef"} err="failed to get container status \"a466a3757af60e6e4f9abc678ae234e4c5894b9af7d321088f4e34d4ed6379ef\": rpc error: code = NotFound desc = could not find container \"a466a3757af60e6e4f9abc678ae234e4c5894b9af7d321088f4e34d4ed6379ef\": container with ID starting with a466a3757af60e6e4f9abc678ae234e4c5894b9af7d321088f4e34d4ed6379ef not found: ID does not exist"
Nov 03 20:20:46.641326 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.641307    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx"]
Nov 03 20:20:46.645474 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.645449    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx"]
Nov 03 20:20:46.892562 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.892459    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1127f5-6d95cf796c-gqdbx" podUID="7f7b664e-2b08-458d-bd6f-324ff24f8441" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.107:50052\" within 1s: context deadline exceeded"
Nov 03 20:20:46.892562 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:46.892477    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-b-1127f5-5b6dfb8c64-jpv4g" podUID="c8adedde-7209-409b-a289-aec02b4e6add" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.108:50052\" within 1s: context deadline exceeded"
Nov 03 20:20:46.892794 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:20:46.892565    4421 logging.go:55] [core] [Channel #1186 SubChannel #1187]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.108:50052", ServerName: "10.134.0.108:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.108:50052: operation was canceled"
Nov 03 20:20:47.252624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.228818    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944"]
Nov 03 20:20:47.252624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.229836    4421 cpu_manager.go:414] "containerMap: removing container" podUID="7f7b664e-2b08-458d-bd6f-324ff24f8441" containerName="payload-pre-processing"
Nov 03 20:20:47.252624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.229857    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f7b664e-2b08-458d-bd6f-324ff24f8441" containerName="payload-pre-processing"
Nov 03 20:20:47.252624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.229898    4421 cpu_manager.go:414] "containerMap: removing container" podUID="c8adedde-7209-409b-a289-aec02b4e6add" containerName="payload-processing"
Nov 03 20:20:47.252624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.229906    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8adedde-7209-409b-a289-aec02b4e6add" containerName="payload-processing"
Nov 03 20:20:47.252624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.230093    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f7b664e-2b08-458d-bd6f-324ff24f8441" containerName="payload-pre-processing"
Nov 03 20:20:47.252624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.230111    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8adedde-7209-409b-a289-aec02b4e6add" containerName="payload-processing"
Nov 03 20:20:47.560333 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.560253    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944"
Nov 03 20:20:47.574886 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.574858    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b07e26\""
Nov 03 20:20:47.575292 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.575270    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b07e26-dockercfg-52vpm\""
Nov 03 20:20:47.584770 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.584746    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944"]
Nov 03 20:20:47.592350 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.592328    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn"]
Nov 03 20:20:47.596436 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.596417    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn"
Nov 03 20:20:47.597935 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.597913    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn"]
Nov 03 20:20:47.679976 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.679942    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3b66ce-9898-4ae3-9a46-8a96365bf82e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn\" (UID: \"2d3b66ce-9898-4ae3-9a46-8a96365bf82e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn"
Nov 03 20:20:47.680140 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.680004    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efca6c1d-0131-4564-a69d-0fd2fa7d4e0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944\" (UID: \"efca6c1d-0131-4564-a69d-0fd2fa7d4e0d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944"
Nov 03 20:20:47.680188 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.680153    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c4mn\" (UniqueName: \"kubernetes.io/projected/efca6c1d-0131-4564-a69d-0fd2fa7d4e0d-kube-api-access-8c4mn\") pod \"payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944\" (UID: \"efca6c1d-0131-4564-a69d-0fd2fa7d4e0d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944"
Nov 03 20:20:47.680243 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.680229    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f74s\" (UniqueName: \"kubernetes.io/projected/2d3b66ce-9898-4ae3-9a46-8a96365bf82e-kube-api-access-4f74s\") pod \"payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn\" (UID: \"2d3b66ce-9898-4ae3-9a46-8a96365bf82e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn"
Nov 03 20:20:47.782062 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.782029    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f74s\" (UniqueName: \"kubernetes.io/projected/2d3b66ce-9898-4ae3-9a46-8a96365bf82e-kube-api-access-4f74s\") pod \"payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn\" (UID: \"2d3b66ce-9898-4ae3-9a46-8a96365bf82e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn"
Nov 03 20:20:47.782237 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.782204    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3b66ce-9898-4ae3-9a46-8a96365bf82e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn\" (UID: \"2d3b66ce-9898-4ae3-9a46-8a96365bf82e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn"
Nov 03 20:20:47.782300 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.782252    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efca6c1d-0131-4564-a69d-0fd2fa7d4e0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944\" (UID: \"efca6c1d-0131-4564-a69d-0fd2fa7d4e0d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944"
Nov 03 20:20:47.782372 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.782350    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c4mn\" (UniqueName: \"kubernetes.io/projected/efca6c1d-0131-4564-a69d-0fd2fa7d4e0d-kube-api-access-8c4mn\") pod \"payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944\" (UID: \"efca6c1d-0131-4564-a69d-0fd2fa7d4e0d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944"
Nov 03 20:20:47.783080 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.783054    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3b66ce-9898-4ae3-9a46-8a96365bf82e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn\" (UID: \"2d3b66ce-9898-4ae3-9a46-8a96365bf82e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn"
Nov 03 20:20:47.783198 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.783088    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efca6c1d-0131-4564-a69d-0fd2fa7d4e0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944\" (UID: \"efca6c1d-0131-4564-a69d-0fd2fa7d4e0d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944"
Nov 03 20:20:47.795095 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.795067    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f74s\" (UniqueName: \"kubernetes.io/projected/2d3b66ce-9898-4ae3-9a46-8a96365bf82e-kube-api-access-4f74s\") pod \"payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn\" (UID: \"2d3b66ce-9898-4ae3-9a46-8a96365bf82e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn"
Nov 03 20:20:47.795330 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.795306    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c4mn\" (UniqueName: \"kubernetes.io/projected/efca6c1d-0131-4564-a69d-0fd2fa7d4e0d-kube-api-access-8c4mn\") pod \"payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944\" (UID: \"efca6c1d-0131-4564-a69d-0fd2fa7d4e0d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944"
Nov 03 20:20:47.892507 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.892415    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944"
Nov 03 20:20:47.915625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:47.914777    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn"
Nov 03 20:20:48.057096 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:48.051463    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f7b664e-2b08-458d-bd6f-324ff24f8441" path="/var/lib/kubelet/pods/7f7b664e-2b08-458d-bd6f-324ff24f8441/volumes"
Nov 03 20:20:48.057096 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:48.052487    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8adedde-7209-409b-a289-aec02b4e6add" path="/var/lib/kubelet/pods/c8adedde-7209-409b-a289-aec02b4e6add/volumes"
Nov 03 20:20:48.296630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:48.294293    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944"]
Nov 03 20:20:48.674626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:48.667770    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn"]
Nov 03 20:20:48.773942 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:20:48.773907    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d3b66ce_9898_4ae3_9a46_8a96365bf82e.slice/crio-bba0dc3db5d43c6b3fb5b7a55f4dffa29993ed260274a377bcbac1e7fb51ed50 WatchSource:0}: Error finding container bba0dc3db5d43c6b3fb5b7a55f4dffa29993ed260274a377bcbac1e7fb51ed50: Status 404 returned error can't find the container with id bba0dc3db5d43c6b3fb5b7a55f4dffa29993ed260274a377bcbac1e7fb51ed50
Nov 03 20:20:48.855174 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:48.833645    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944" event={"ID":"efca6c1d-0131-4564-a69d-0fd2fa7d4e0d","Type":"ContainerStarted","Data":"d068e4e68d9a7f2c809c7c1c8b8a9937c9bd82af3812180ceef4660f1b97e35b"}
Nov 03 20:20:49.392569 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:49.392390    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82"
Nov 03 20:20:49.681783 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:20:49.681687    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:20:49.841811 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:49.841772    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn" event={"ID":"2d3b66ce-9898-4ae3-9a46-8a96365bf82e","Type":"ContainerStarted","Data":"5fb3c286aa2f135b3b5ae743cb0522c5b5b8ccd624f63359831ebffe7d7b797b"}
Nov 03 20:20:49.841993 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:49.841835    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn" event={"ID":"2d3b66ce-9898-4ae3-9a46-8a96365bf82e","Type":"ContainerStarted","Data":"bba0dc3db5d43c6b3fb5b7a55f4dffa29993ed260274a377bcbac1e7fb51ed50"}
Nov 03 20:20:49.841993 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:49.841864    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn"
Nov 03 20:20:49.843407 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:49.843384    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944" event={"ID":"efca6c1d-0131-4564-a69d-0fd2fa7d4e0d","Type":"ContainerStarted","Data":"af3a7ad3ec1fc0eb6bbbc81b0e8f5572b2268ad4c74e5c38d0229a984e80e755"}
Nov 03 20:20:49.843532 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:49.843463    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944"
Nov 03 20:20:49.935840 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:49.910200    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn" podStartSLOduration=2.910177062 podStartE2EDuration="2.910177062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:49.875276714 +0000 UTC m=+1180.008405177" watchObservedRunningTime="2026-11-03 20:20:49.910177062 +0000 UTC m=+1180.043305525"
Nov 03 20:20:49.962995 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:49.962933    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944" podStartSLOduration=2.962900995 podStartE2EDuration="2.962900995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:49.936147186 +0000 UTC m=+1180.069275661" watchObservedRunningTime="2026-11-03 20:20:49.962900995 +0000 UTC m=+1180.096029459"
Nov 03 20:20:52.599338 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:52.599303    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s"]
Nov 03 20:20:52.602734 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:52.602716    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s"
Nov 03 20:20:52.612914 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:52.612890    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d6789f\""
Nov 03 20:20:52.615917 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:52.615890    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s"]
Nov 03 20:20:52.772625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:52.760515    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db876db-2b38-4d10-88ab-4302077c072b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s\" (UID: \"3db876db-2b38-4d10-88ab-4302077c072b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s"
Nov 03 20:20:52.772625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:52.760766    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hzqd\" (UniqueName: \"kubernetes.io/projected/3db876db-2b38-4d10-88ab-4302077c072b-kube-api-access-8hzqd\") pod \"payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s\" (UID: \"3db876db-2b38-4d10-88ab-4302077c072b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s"
Nov 03 20:20:52.870678 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:52.862251    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db876db-2b38-4d10-88ab-4302077c072b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s\" (UID: \"3db876db-2b38-4d10-88ab-4302077c072b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s"
Nov 03 20:20:52.870678 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:52.862428    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hzqd\" (UniqueName: \"kubernetes.io/projected/3db876db-2b38-4d10-88ab-4302077c072b-kube-api-access-8hzqd\") pod \"payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s\" (UID: \"3db876db-2b38-4d10-88ab-4302077c072b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s"
Nov 03 20:20:52.870678 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:52.870617    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db876db-2b38-4d10-88ab-4302077c072b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s\" (UID: \"3db876db-2b38-4d10-88ab-4302077c072b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s"
Nov 03 20:20:52.975621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:52.973339    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hzqd\" (UniqueName: \"kubernetes.io/projected/3db876db-2b38-4d10-88ab-4302077c072b-kube-api-access-8hzqd\") pod \"payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s\" (UID: \"3db876db-2b38-4d10-88ab-4302077c072b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s"
Nov 03 20:20:53.219552 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:53.219516    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s"
Nov 03 20:20:53.608592 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:53.608565    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s"]
Nov 03 20:20:53.608986 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:20:53.608773    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3db876db_2b38_4d10_88ab_4302077c072b.slice/crio-1d3dd6027f993d22cdbd8e778c7c5b3ed08c15c0285f807b28e50f6abdc62248 WatchSource:0}: Error finding container 1d3dd6027f993d22cdbd8e778c7c5b3ed08c15c0285f807b28e50f6abdc62248: Status 404 returned error can't find the container with id 1d3dd6027f993d22cdbd8e778c7c5b3ed08c15c0285f807b28e50f6abdc62248
Nov 03 20:20:53.839714 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:53.839687    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85"
Nov 03 20:20:54.093022 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:54.092759    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s"
Nov 03 20:20:54.093022 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:54.092849    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s" event={"ID":"3db876db-2b38-4d10-88ab-4302077c072b","Type":"ContainerStarted","Data":"b35b660c15c57ec49036e421eb72b1a88adb2a20f2c8660386e81d0efc68a97e"}
Nov 03 20:20:54.093022 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:54.092892    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s" event={"ID":"3db876db-2b38-4d10-88ab-4302077c072b","Type":"ContainerStarted","Data":"1d3dd6027f993d22cdbd8e778c7c5b3ed08c15c0285f807b28e50f6abdc62248"}
Nov 03 20:20:54.099084 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:54.098983    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s" podStartSLOduration=2.098962596 podStartE2EDuration="2.098962596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:54.095793498 +0000 UTC m=+1184.228921961" watchObservedRunningTime="2026-11-03 20:20:54.098962596 +0000 UTC m=+1184.232091059"
Nov 03 20:20:55.924087 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:55.924042    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq"]
Nov 03 20:20:55.938806 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:55.938769    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq"
Nov 03 20:20:55.941766 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:55.941732    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq"]
Nov 03 20:20:55.942864 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:55.942846    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b07e26-serving-cert\""
Nov 03 20:20:55.996044 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:55.995997    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25d78e0c-9cfd-46e0-a6cd-a060f352b860-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq\" (UID: \"25d78e0c-9cfd-46e0-a6cd-a060f352b860\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq"
Nov 03 20:20:55.996208 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:55.996141    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25d78e0c-9cfd-46e0-a6cd-a060f352b860-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq\" (UID: \"25d78e0c-9cfd-46e0-a6cd-a060f352b860\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq"
Nov 03 20:20:55.996292 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:55.996264    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25d78e0c-9cfd-46e0-a6cd-a060f352b860-metrics-certs\") pod \"maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq\" (UID: \"25d78e0c-9cfd-46e0-a6cd-a060f352b860\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq"
Nov 03 20:20:55.996520 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:55.996502    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xbq2\" (UniqueName: \"kubernetes.io/projected/25d78e0c-9cfd-46e0-a6cd-a060f352b860-kube-api-access-2xbq2\") pod \"maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq\" (UID: \"25d78e0c-9cfd-46e0-a6cd-a060f352b860\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq"
Nov 03 20:20:56.108626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:56.105135    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xbq2\" (UniqueName: \"kubernetes.io/projected/25d78e0c-9cfd-46e0-a6cd-a060f352b860-kube-api-access-2xbq2\") pod \"maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq\" (UID: \"25d78e0c-9cfd-46e0-a6cd-a060f352b860\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq"
Nov 03 20:20:56.108626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:56.105204    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25d78e0c-9cfd-46e0-a6cd-a060f352b860-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq\" (UID: \"25d78e0c-9cfd-46e0-a6cd-a060f352b860\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq"
Nov 03 20:20:56.108626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:56.105336    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25d78e0c-9cfd-46e0-a6cd-a060f352b860-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq\" (UID: \"25d78e0c-9cfd-46e0-a6cd-a060f352b860\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq"
Nov 03 20:20:56.108626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:56.105482    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25d78e0c-9cfd-46e0-a6cd-a060f352b860-metrics-certs\") pod \"maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq\" (UID: \"25d78e0c-9cfd-46e0-a6cd-a060f352b860\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq"
Nov 03 20:20:56.118637 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:56.111392    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25d78e0c-9cfd-46e0-a6cd-a060f352b860-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq\" (UID: \"25d78e0c-9cfd-46e0-a6cd-a060f352b860\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq"
Nov 03 20:20:56.159655 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:56.158379    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25d78e0c-9cfd-46e0-a6cd-a060f352b860-metrics-certs\") pod \"maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq\" (UID: \"25d78e0c-9cfd-46e0-a6cd-a060f352b860\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq"
Nov 03 20:20:56.159655 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:56.158958    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25d78e0c-9cfd-46e0-a6cd-a060f352b860-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq\" (UID: \"25d78e0c-9cfd-46e0-a6cd-a060f352b860\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq"
Nov 03 20:20:56.185663 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:56.179418    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xbq2\" (UniqueName: \"kubernetes.io/projected/25d78e0c-9cfd-46e0-a6cd-a060f352b860-kube-api-access-2xbq2\") pod \"maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq\" (UID: \"25d78e0c-9cfd-46e0-a6cd-a060f352b860\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq"
Nov 03 20:20:56.265463 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:56.265428    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq"
Nov 03 20:20:56.681869 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:56.680183    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq"]
Nov 03 20:20:57.203298 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:57.191874    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq" event={"ID":"25d78e0c-9cfd-46e0-a6cd-a060f352b860","Type":"ContainerStarted","Data":"1a323a0c1a263d12f8fd076cabc89a2b1cfee21c17b15a97dcc58105bbf14a7f"}
Nov 03 20:20:58.222685 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:58.220806    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq" event={"ID":"25d78e0c-9cfd-46e0-a6cd-a060f352b860","Type":"ContainerStarted","Data":"aab65d8778267504af58a483aa46d9979ca1b41764bd2837fdb51e8bf76c119a"}
Nov 03 20:20:58.222685 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:20:58.222329    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq"
Nov 03 20:21:00.859919 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:00.859667    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn"
Nov 03 20:21:00.869766 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:00.869735    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944"
Nov 03 20:21:00.908499 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:00.908439    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq" podStartSLOduration=5.415739288 podStartE2EDuration="5.908422993s" totalImagesPullingTime="492.683705ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:58.260576618 +0000 UTC m=+1188.393705103" watchObservedRunningTime="2026-11-03 20:21:00.908422993 +0000 UTC m=+1191.041551454"
Nov 03 20:21:05.037762 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:05.037729    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s"
Nov 03 20:21:05.241801 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:05.241771    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq"
Nov 03 20:21:06.413154 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:06.413115    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb"]
Nov 03 20:21:06.420455 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:06.420431    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb"
Nov 03 20:21:06.436973 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:06.436947    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-bf9b0a-serving-cert\""
Nov 03 20:21:06.445497 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:06.445471    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb"]
Nov 03 20:21:06.575440 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:06.575339    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lrtj\" (UniqueName: \"kubernetes.io/projected/1007c757-280e-4436-8c9f-e5ac5deda164-kube-api-access-2lrtj\") pod \"maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb\" (UID: \"1007c757-280e-4436-8c9f-e5ac5deda164\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb"
Nov 03 20:21:06.580336 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:06.580279    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1007c757-280e-4436-8c9f-e5ac5deda164-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb\" (UID: \"1007c757-280e-4436-8c9f-e5ac5deda164\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb"
Nov 03 20:21:06.580743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:06.580723    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1007c757-280e-4436-8c9f-e5ac5deda164-metrics-certs\") pod \"maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb\" (UID: \"1007c757-280e-4436-8c9f-e5ac5deda164\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb"
Nov 03 20:21:06.581163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:06.581096    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1007c757-280e-4436-8c9f-e5ac5deda164-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb\" (UID: \"1007c757-280e-4436-8c9f-e5ac5deda164\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb"
Nov 03 20:21:06.683076 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:06.682991    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1007c757-280e-4436-8c9f-e5ac5deda164-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb\" (UID: \"1007c757-280e-4436-8c9f-e5ac5deda164\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb"
Nov 03 20:21:06.683076 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:06.683054    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1007c757-280e-4436-8c9f-e5ac5deda164-metrics-certs\") pod \"maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb\" (UID: \"1007c757-280e-4436-8c9f-e5ac5deda164\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb"
Nov 03 20:21:06.683309 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:06.683096    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1007c757-280e-4436-8c9f-e5ac5deda164-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb\" (UID: \"1007c757-280e-4436-8c9f-e5ac5deda164\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb"
Nov 03 20:21:06.683309 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:06.683199    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lrtj\" (UniqueName: \"kubernetes.io/projected/1007c757-280e-4436-8c9f-e5ac5deda164-kube-api-access-2lrtj\") pod \"maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb\" (UID: \"1007c757-280e-4436-8c9f-e5ac5deda164\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb"
Nov 03 20:21:06.683831 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:06.683812    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1007c757-280e-4436-8c9f-e5ac5deda164-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb\" (UID: \"1007c757-280e-4436-8c9f-e5ac5deda164\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb"
Nov 03 20:21:06.685964 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:06.685941    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1007c757-280e-4436-8c9f-e5ac5deda164-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb\" (UID: \"1007c757-280e-4436-8c9f-e5ac5deda164\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb"
Nov 03 20:21:06.687294 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:06.687274    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1007c757-280e-4436-8c9f-e5ac5deda164-metrics-certs\") pod \"maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb\" (UID: \"1007c757-280e-4436-8c9f-e5ac5deda164\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb"
Nov 03 20:21:06.695969 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:06.695949    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lrtj\" (UniqueName: \"kubernetes.io/projected/1007c757-280e-4436-8c9f-e5ac5deda164-kube-api-access-2lrtj\") pod \"maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb\" (UID: \"1007c757-280e-4436-8c9f-e5ac5deda164\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb"
Nov 03 20:21:06.746205 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:06.746179    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb"
Nov 03 20:21:07.043918 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:07.043889    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb"]
Nov 03 20:21:07.047395 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:21:07.047354    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1007c757_280e_4436_8c9f_e5ac5deda164.slice/crio-4a3771ae8fd95257bd8a30e3d634ee6103333e8d4b865b78eeb3ccdc6660939f WatchSource:0}: Error finding container 4a3771ae8fd95257bd8a30e3d634ee6103333e8d4b865b78eeb3ccdc6660939f: Status 404 returned error can't find the container with id 4a3771ae8fd95257bd8a30e3d634ee6103333e8d4b865b78eeb3ccdc6660939f
Nov 03 20:21:07.417742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:07.403782    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb" event={"ID":"1007c757-280e-4436-8c9f-e5ac5deda164","Type":"ContainerStarted","Data":"4a3771ae8fd95257bd8a30e3d634ee6103333e8d4b865b78eeb3ccdc6660939f"}
Nov 03 20:21:08.414243 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:08.414193    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb" event={"ID":"1007c757-280e-4436-8c9f-e5ac5deda164","Type":"ContainerStarted","Data":"9414ca6a6aa63a7151776550aa5de690d4e14c009c5d23efe5f3fd2eb5762b45"}
Nov 03 20:21:08.414447 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:08.414292    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb"
Nov 03 20:21:08.439360 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:08.439305    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb" podStartSLOduration=1.8618691809999999 podStartE2EDuration="2.439290935s" totalImagesPullingTime="577.421754ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:08.437320772 +0000 UTC m=+1198.570449238" watchObservedRunningTime="2026-11-03 20:21:08.439290935 +0000 UTC m=+1198.572419398"
Nov 03 20:21:10.872227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:10.872128    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn"]
Nov 03 20:21:10.882893 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:10.882863    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944"]
Nov 03 20:21:14.485794 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:14.483773    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb"
Nov 03 20:21:15.873055 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:15.873006    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn" podUID="2d3b66ce-9898-4ae3-9a46-8a96365bf82e" containerName="payload-processing" containerID="cri-o://5fb3c286aa2f135b3b5ae743cb0522c5b5b8ccd624f63359831ebffe7d7b797b" gracePeriod=25
Nov 03 20:21:15.884020 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:15.883967    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944" podUID="efca6c1d-0131-4564-a69d-0fd2fa7d4e0d" containerName="payload-pre-processing" containerID="cri-o://af3a7ad3ec1fc0eb6bbbc81b0e8f5572b2268ad4c74e5c38d0229a984e80e755" gracePeriod=25
Nov 03 20:21:16.534084 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:16.533910    4421 generic.go:356] "Generic (PLEG): container finished" podID="2d3b66ce-9898-4ae3-9a46-8a96365bf82e" containerID="5fb3c286aa2f135b3b5ae743cb0522c5b5b8ccd624f63359831ebffe7d7b797b" exitCode=0
Nov 03 20:21:16.534084 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:16.534034    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn" event={"ID":"2d3b66ce-9898-4ae3-9a46-8a96365bf82e","Type":"ContainerDied","Data":"5fb3c286aa2f135b3b5ae743cb0522c5b5b8ccd624f63359831ebffe7d7b797b"}
Nov 03 20:21:16.536012 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:16.535909    4421 generic.go:356] "Generic (PLEG): container finished" podID="efca6c1d-0131-4564-a69d-0fd2fa7d4e0d" containerID="af3a7ad3ec1fc0eb6bbbc81b0e8f5572b2268ad4c74e5c38d0229a984e80e755" exitCode=0
Nov 03 20:21:16.536407 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:16.536035    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944" event={"ID":"efca6c1d-0131-4564-a69d-0fd2fa7d4e0d","Type":"ContainerDied","Data":"af3a7ad3ec1fc0eb6bbbc81b0e8f5572b2268ad4c74e5c38d0229a984e80e755"}
Nov 03 20:21:16.801684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:16.784479    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn"
Nov 03 20:21:16.917215 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:16.917189    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944"
Nov 03 20:21:16.999210 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:16.999174    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d3b66ce-9898-4ae3-9a46-8a96365bf82e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3b66ce-9898-4ae3-9a46-8a96365bf82e-plugins-config-volume\") pod \"2d3b66ce-9898-4ae3-9a46-8a96365bf82e\" (UID: \"2d3b66ce-9898-4ae3-9a46-8a96365bf82e\") "
Nov 03 20:21:16.999494 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:16.999476    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d3b66ce-9898-4ae3-9a46-8a96365bf82e-kube-api-access-4f74s\" (UniqueName: \"kubernetes.io/projected/2d3b66ce-9898-4ae3-9a46-8a96365bf82e-kube-api-access-4f74s\") pod \"2d3b66ce-9898-4ae3-9a46-8a96365bf82e\" (UID: \"2d3b66ce-9898-4ae3-9a46-8a96365bf82e\") "
Nov 03 20:21:16.999935 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:16.999902    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d3b66ce-9898-4ae3-9a46-8a96365bf82e-plugins-config-volume" pod "2d3b66ce-9898-4ae3-9a46-8a96365bf82e" (UID: "2d3b66ce-9898-4ae3-9a46-8a96365bf82e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:17.000237 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:17.000215    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efca6c1d-0131-4564-a69d-0fd2fa7d4e0d-kube-api-access-8c4mn\" (UniqueName: \"kubernetes.io/projected/efca6c1d-0131-4564-a69d-0fd2fa7d4e0d-kube-api-access-8c4mn\") pod \"efca6c1d-0131-4564-a69d-0fd2fa7d4e0d\" (UID: \"efca6c1d-0131-4564-a69d-0fd2fa7d4e0d\") "
Nov 03 20:21:17.000457 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:17.000437    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efca6c1d-0131-4564-a69d-0fd2fa7d4e0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efca6c1d-0131-4564-a69d-0fd2fa7d4e0d-plugins-config-volume\") pod \"efca6c1d-0131-4564-a69d-0fd2fa7d4e0d\" (UID: \"efca6c1d-0131-4564-a69d-0fd2fa7d4e0d\") "
Nov 03 20:21:17.001387 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:17.001362    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d3b66ce-9898-4ae3-9a46-8a96365bf82e-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:17.001904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:17.001880    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efca6c1d-0131-4564-a69d-0fd2fa7d4e0d-plugins-config-volume" pod "efca6c1d-0131-4564-a69d-0fd2fa7d4e0d" (UID: "efca6c1d-0131-4564-a69d-0fd2fa7d4e0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:17.003704 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:17.003575    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d3b66ce-9898-4ae3-9a46-8a96365bf82e-kube-api-access-4f74s" pod "2d3b66ce-9898-4ae3-9a46-8a96365bf82e" (UID: "2d3b66ce-9898-4ae3-9a46-8a96365bf82e"). InnerVolumeSpecName "kube-api-access-4f74s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:17.008167 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:17.008142    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efca6c1d-0131-4564-a69d-0fd2fa7d4e0d-kube-api-access-8c4mn" pod "efca6c1d-0131-4564-a69d-0fd2fa7d4e0d" (UID: "efca6c1d-0131-4564-a69d-0fd2fa7d4e0d"). InnerVolumeSpecName "kube-api-access-8c4mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:17.103374 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:17.103299    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f74s\" (UniqueName: \"kubernetes.io/projected/2d3b66ce-9898-4ae3-9a46-8a96365bf82e-kube-api-access-4f74s\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:17.103374 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:17.103330    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c4mn\" (UniqueName: \"kubernetes.io/projected/efca6c1d-0131-4564-a69d-0fd2fa7d4e0d-kube-api-access-8c4mn\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:17.103374 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:17.103338    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efca6c1d-0131-4564-a69d-0fd2fa7d4e0d-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:17.591625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:17.589260    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn" event={"ID":"2d3b66ce-9898-4ae3-9a46-8a96365bf82e","Type":"ContainerDied","Data":"bba0dc3db5d43c6b3fb5b7a55f4dffa29993ed260274a377bcbac1e7fb51ed50"}
Nov 03 20:21:17.591625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:17.589335    4421 scope.go:122] "RemoveContainer" containerID="5fb3c286aa2f135b3b5ae743cb0522c5b5b8ccd624f63359831ebffe7d7b797b"
Nov 03 20:21:17.591625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:17.589484    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn"
Nov 03 20:21:17.605847 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:17.593346    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944" event={"ID":"efca6c1d-0131-4564-a69d-0fd2fa7d4e0d","Type":"ContainerDied","Data":"d068e4e68d9a7f2c809c7c1c8b8a9937c9bd82af3812180ceef4660f1b97e35b"}
Nov 03 20:21:17.605847 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:17.593442    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944"
Nov 03 20:21:17.614261 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:17.613904    4421 scope.go:122] "RemoveContainer" containerID="af3a7ad3ec1fc0eb6bbbc81b0e8f5572b2268ad4c74e5c38d0229a984e80e755"
Nov 03 20:21:17.633566 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:17.633546    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944"]
Nov 03 20:21:17.639648 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:17.639625    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b07e26-74757d58b7-mh944"]
Nov 03 20:21:17.699623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:17.698632    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn"]
Nov 03 20:21:17.703320 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:17.703285    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b07e26-79f5c4967d-q8fdn"]
Nov 03 20:21:18.023300 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:18.023270    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d3b66ce-9898-4ae3-9a46-8a96365bf82e" path="/var/lib/kubelet/pods/2d3b66ce-9898-4ae3-9a46-8a96365bf82e/volumes"
Nov 03 20:21:18.023845 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:18.023826    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efca6c1d-0131-4564-a69d-0fd2fa7d4e0d" path="/var/lib/kubelet/pods/efca6c1d-0131-4564-a69d-0fd2fa7d4e0d/volumes"
Nov 03 20:21:19.633152 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:21:19.632984    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:21:20.016781 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:20.016713    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc"]
Nov 03 20:21:20.017772 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:20.017738    4421 cpu_manager.go:414] "containerMap: removing container" podUID="efca6c1d-0131-4564-a69d-0fd2fa7d4e0d" containerName="payload-pre-processing"
Nov 03 20:21:20.017772 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:20.017772    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efca6c1d-0131-4564-a69d-0fd2fa7d4e0d" containerName="payload-pre-processing"
Nov 03 20:21:20.017964 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:20.017823    4421 cpu_manager.go:414] "containerMap: removing container" podUID="2d3b66ce-9898-4ae3-9a46-8a96365bf82e" containerName="payload-processing"
Nov 03 20:21:20.017964 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:20.017830    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d3b66ce-9898-4ae3-9a46-8a96365bf82e" containerName="payload-processing"
Nov 03 20:21:20.018069 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:20.018014    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="efca6c1d-0131-4564-a69d-0fd2fa7d4e0d" containerName="payload-pre-processing"
Nov 03 20:21:20.018069 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:20.018036    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d3b66ce-9898-4ae3-9a46-8a96365bf82e" containerName="payload-processing"
Nov 03 20:21:20.025652 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:20.022689    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc"]
Nov 03 20:21:20.025652 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:20.022877    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc"
Nov 03 20:21:20.026742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:20.026691    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-400777-serving-cert\""
Nov 03 20:21:20.136947 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:20.135940    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c38927-79b7-4557-8909-9e792fef6a8c-openshift-service-ca\") pod \"maas-api-e2e-api-b-400777-7d86648cd5-wh9wc\" (UID: \"63c38927-79b7-4557-8909-9e792fef6a8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc"
Nov 03 20:21:20.136947 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:20.136143    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c38927-79b7-4557-8909-9e792fef6a8c-metrics-certs\") pod \"maas-api-e2e-api-b-400777-7d86648cd5-wh9wc\" (UID: \"63c38927-79b7-4557-8909-9e792fef6a8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc"
Nov 03 20:21:20.136947 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:20.136230    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98ndx\" (UniqueName: \"kubernetes.io/projected/63c38927-79b7-4557-8909-9e792fef6a8c-kube-api-access-98ndx\") pod \"maas-api-e2e-api-b-400777-7d86648cd5-wh9wc\" (UID: \"63c38927-79b7-4557-8909-9e792fef6a8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc"
Nov 03 20:21:20.136947 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:20.136321    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c38927-79b7-4557-8909-9e792fef6a8c-maas-api-tls\") pod \"maas-api-e2e-api-b-400777-7d86648cd5-wh9wc\" (UID: \"63c38927-79b7-4557-8909-9e792fef6a8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc"
Nov 03 20:21:20.241622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:20.240992    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c38927-79b7-4557-8909-9e792fef6a8c-openshift-service-ca\") pod \"maas-api-e2e-api-b-400777-7d86648cd5-wh9wc\" (UID: \"63c38927-79b7-4557-8909-9e792fef6a8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc"
Nov 03 20:21:20.241622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:20.241188    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c38927-79b7-4557-8909-9e792fef6a8c-metrics-certs\") pod \"maas-api-e2e-api-b-400777-7d86648cd5-wh9wc\" (UID: \"63c38927-79b7-4557-8909-9e792fef6a8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc"
Nov 03 20:21:20.241622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:20.241268    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98ndx\" (UniqueName: \"kubernetes.io/projected/63c38927-79b7-4557-8909-9e792fef6a8c-kube-api-access-98ndx\") pod \"maas-api-e2e-api-b-400777-7d86648cd5-wh9wc\" (UID: \"63c38927-79b7-4557-8909-9e792fef6a8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc"
Nov 03 20:21:20.241622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:20.241362    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c38927-79b7-4557-8909-9e792fef6a8c-maas-api-tls\") pod \"maas-api-e2e-api-b-400777-7d86648cd5-wh9wc\" (UID: \"63c38927-79b7-4557-8909-9e792fef6a8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc"
Nov 03 20:21:20.241999 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:21:20.241834    4421 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-400777-serving-cert: secret "maas-api-e2e-api-b-400777-serving-cert" not found
Nov 03 20:21:20.241999 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:21:20.241931    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63c38927-79b7-4557-8909-9e792fef6a8c-maas-api-tls podName:63c38927-79b7-4557-8909-9e792fef6a8c nodeName:}" failed. No retries permitted until 2026-11-03 20:21:20.741893903 +0000 UTC m=+1210.875022362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/63c38927-79b7-4557-8909-9e792fef6a8c-maas-api-tls") pod "maas-api-e2e-api-b-400777-7d86648cd5-wh9wc" (UID: "63c38927-79b7-4557-8909-9e792fef6a8c") : secret "maas-api-e2e-api-b-400777-serving-cert" not found
Nov 03 20:21:20.269258 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:20.257181    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c38927-79b7-4557-8909-9e792fef6a8c-openshift-service-ca\") pod \"maas-api-e2e-api-b-400777-7d86648cd5-wh9wc\" (UID: \"63c38927-79b7-4557-8909-9e792fef6a8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc"
Nov 03 20:21:20.269258 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:20.267121    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c38927-79b7-4557-8909-9e792fef6a8c-metrics-certs\") pod \"maas-api-e2e-api-b-400777-7d86648cd5-wh9wc\" (UID: \"63c38927-79b7-4557-8909-9e792fef6a8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc"
Nov 03 20:21:20.287618 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:20.271212    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98ndx\" (UniqueName: \"kubernetes.io/projected/63c38927-79b7-4557-8909-9e792fef6a8c-kube-api-access-98ndx\") pod \"maas-api-e2e-api-b-400777-7d86648cd5-wh9wc\" (UID: \"63c38927-79b7-4557-8909-9e792fef6a8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc"
Nov 03 20:21:20.782679 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:20.781920    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c38927-79b7-4557-8909-9e792fef6a8c-maas-api-tls\") pod \"maas-api-e2e-api-b-400777-7d86648cd5-wh9wc\" (UID: \"63c38927-79b7-4557-8909-9e792fef6a8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc"
Nov 03 20:21:20.782679 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:21:20.782158    4421 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-400777-serving-cert: secret "maas-api-e2e-api-b-400777-serving-cert" not found
Nov 03 20:21:20.782679 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:21:20.782243    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63c38927-79b7-4557-8909-9e792fef6a8c-maas-api-tls podName:63c38927-79b7-4557-8909-9e792fef6a8c nodeName:}" failed. No retries permitted until 2026-11-03 20:21:21.782223944 +0000 UTC m=+1211.915352398 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/63c38927-79b7-4557-8909-9e792fef6a8c-maas-api-tls") pod "maas-api-e2e-api-b-400777-7d86648cd5-wh9wc" (UID: "63c38927-79b7-4557-8909-9e792fef6a8c") : secret "maas-api-e2e-api-b-400777-serving-cert" not found
Nov 03 20:21:21.579624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:21.579274    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s"]
Nov 03 20:21:21.948692 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:21.933187    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c38927-79b7-4557-8909-9e792fef6a8c-maas-api-tls\") pod \"maas-api-e2e-api-b-400777-7d86648cd5-wh9wc\" (UID: \"63c38927-79b7-4557-8909-9e792fef6a8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc"
Nov 03 20:21:22.049624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:22.048543    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c38927-79b7-4557-8909-9e792fef6a8c-maas-api-tls\") pod \"maas-api-e2e-api-b-400777-7d86648cd5-wh9wc\" (UID: \"63c38927-79b7-4557-8909-9e792fef6a8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc"
Nov 03 20:21:22.222810 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:22.222717    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc"
Nov 03 20:21:22.670205 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:21:22.668945    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c38927_79b7_4557_8909_9e792fef6a8c.slice/crio-1a49ec05fb8648ebdf6ffda66a6a55690240b31d1cecc78f83f5ce89d5e0c18e WatchSource:0}: Error finding container 1a49ec05fb8648ebdf6ffda66a6a55690240b31d1cecc78f83f5ce89d5e0c18e: Status 404 returned error can't find the container with id 1a49ec05fb8648ebdf6ffda66a6a55690240b31d1cecc78f83f5ce89d5e0c18e
Nov 03 20:21:22.670205 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:22.670058    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc"]
Nov 03 20:21:22.828289 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:22.828249    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc" event={"ID":"63c38927-79b7-4557-8909-9e792fef6a8c","Type":"ContainerStarted","Data":"1a49ec05fb8648ebdf6ffda66a6a55690240b31d1cecc78f83f5ce89d5e0c18e"}
Nov 03 20:21:23.341551 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:23.341466    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6"]
Nov 03 20:21:23.342643 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:23.341714    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6" podUID="f1e67982-68de-4503-af48-8cb554c6679e" containerName="maas-api" containerID="cri-o://d590806b4d3b0ddd26d3ec0025ce3fb0416e1db4206f676d107d24c3a73c9a8f" gracePeriod=30
Nov 03 20:21:23.843893 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:23.843846    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc" event={"ID":"63c38927-79b7-4557-8909-9e792fef6a8c","Type":"ContainerStarted","Data":"9ed973fca5c324503834a93bdb3e99aeeacb75ce942c61cceccb414f47a4edc2"}
Nov 03 20:21:23.845134 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:23.845054    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc"
Nov 03 20:21:23.849521 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:23.847248    4421 generic.go:356] "Generic (PLEG): container finished" podID="f1e67982-68de-4503-af48-8cb554c6679e" containerID="d590806b4d3b0ddd26d3ec0025ce3fb0416e1db4206f676d107d24c3a73c9a8f" exitCode=0
Nov 03 20:21:23.849521 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:23.847305    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6" event={"ID":"f1e67982-68de-4503-af48-8cb554c6679e","Type":"ContainerDied","Data":"d590806b4d3b0ddd26d3ec0025ce3fb0416e1db4206f676d107d24c3a73c9a8f"}
Nov 03 20:21:23.868438 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:23.867993    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc" podStartSLOduration=4.482887664 podStartE2EDuration="4.867979083s" totalImagesPullingTime="385.091419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:23.867257419 +0000 UTC m=+1214.000385881" watchObservedRunningTime="2026-11-03 20:21:23.867979083 +0000 UTC m=+1214.001107545"
Nov 03 20:21:23.883902 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:23.883401    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6"
Nov 03 20:21:23.954456 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:23.954412    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1e67982-68de-4503-af48-8cb554c6679e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1e67982-68de-4503-af48-8cb554c6679e-maas-api-tls\") pod \"f1e67982-68de-4503-af48-8cb554c6679e\" (UID: \"f1e67982-68de-4503-af48-8cb554c6679e\") "
Nov 03 20:21:23.954689 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:23.954669    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1e67982-68de-4503-af48-8cb554c6679e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1e67982-68de-4503-af48-8cb554c6679e-openshift-service-ca\") pod \"f1e67982-68de-4503-af48-8cb554c6679e\" (UID: \"f1e67982-68de-4503-af48-8cb554c6679e\") "
Nov 03 20:21:23.954756 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:23.954710    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1e67982-68de-4503-af48-8cb554c6679e-kube-api-access-jfwtj\" (UniqueName: \"kubernetes.io/projected/f1e67982-68de-4503-af48-8cb554c6679e-kube-api-access-jfwtj\") pod \"f1e67982-68de-4503-af48-8cb554c6679e\" (UID: \"f1e67982-68de-4503-af48-8cb554c6679e\") "
Nov 03 20:21:23.954830 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:23.954809    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1e67982-68de-4503-af48-8cb554c6679e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1e67982-68de-4503-af48-8cb554c6679e-metrics-certs\") pod \"f1e67982-68de-4503-af48-8cb554c6679e\" (UID: \"f1e67982-68de-4503-af48-8cb554c6679e\") "
Nov 03 20:21:23.955098 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:23.955070    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1e67982-68de-4503-af48-8cb554c6679e-openshift-service-ca" pod "f1e67982-68de-4503-af48-8cb554c6679e" (UID: "f1e67982-68de-4503-af48-8cb554c6679e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:23.955750 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:23.955724    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1e67982-68de-4503-af48-8cb554c6679e-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:23.958072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:23.958034    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1e67982-68de-4503-af48-8cb554c6679e-metrics-certs" pod "f1e67982-68de-4503-af48-8cb554c6679e" (UID: "f1e67982-68de-4503-af48-8cb554c6679e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:21:23.958223 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:23.958190    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1e67982-68de-4503-af48-8cb554c6679e-maas-api-tls" pod "f1e67982-68de-4503-af48-8cb554c6679e" (UID: "f1e67982-68de-4503-af48-8cb554c6679e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:21:23.958281 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:23.958248    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e67982-68de-4503-af48-8cb554c6679e-kube-api-access-jfwtj" pod "f1e67982-68de-4503-af48-8cb554c6679e" (UID: "f1e67982-68de-4503-af48-8cb554c6679e"). InnerVolumeSpecName "kube-api-access-jfwtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:24.075219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:24.073486    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1e67982-68de-4503-af48-8cb554c6679e-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:24.075219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:24.073524    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1e67982-68de-4503-af48-8cb554c6679e-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:24.075219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:24.073539    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfwtj\" (UniqueName: \"kubernetes.io/projected/f1e67982-68de-4503-af48-8cb554c6679e-kube-api-access-jfwtj\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:24.827051 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:24.827014    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7"]
Nov 03 20:21:24.828063 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:24.828042    4421 cpu_manager.go:414] "containerMap: removing container" podUID="f1e67982-68de-4503-af48-8cb554c6679e" containerName="maas-api"
Nov 03 20:21:24.828063 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:24.828064    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1e67982-68de-4503-af48-8cb554c6679e" containerName="maas-api"
Nov 03 20:21:24.828318 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:24.828308    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1e67982-68de-4503-af48-8cb554c6679e" containerName="maas-api"
Nov 03 20:21:24.835393 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:24.835363    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7"
Nov 03 20:21:24.847754 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:24.847722    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7"]
Nov 03 20:21:24.853156 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:24.853129    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b07e26\""
Nov 03 20:21:24.874139 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:24.874100    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6" event={"ID":"f1e67982-68de-4503-af48-8cb554c6679e","Type":"ContainerDied","Data":"c8401976e66aef3d1b5912cda01dabbeefcfbda4f1d13810e47837276d7281e8"}
Nov 03 20:21:24.874323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:24.874166    4421 scope.go:122] "RemoveContainer" containerID="d590806b4d3b0ddd26d3ec0025ce3fb0416e1db4206f676d107d24c3a73c9a8f"
Nov 03 20:21:24.874323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:24.874288    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6"
Nov 03 20:21:24.896852 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:24.896776    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drppz\" (UniqueName: \"kubernetes.io/projected/7f3ff7e0-2c72-45c4-bc9c-e58967921176-kube-api-access-drppz\") pod \"payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7\" (UID: \"7f3ff7e0-2c72-45c4-bc9c-e58967921176\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7"
Nov 03 20:21:24.897020 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:24.896933    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3ff7e0-2c72-45c4-bc9c-e58967921176-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7\" (UID: \"7f3ff7e0-2c72-45c4-bc9c-e58967921176\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7"
Nov 03 20:21:24.914174 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:24.914136    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6"]
Nov 03 20:21:24.920832 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:24.920793    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-qv7l6"]
Nov 03 20:21:25.014627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:25.002950    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drppz\" (UniqueName: \"kubernetes.io/projected/7f3ff7e0-2c72-45c4-bc9c-e58967921176-kube-api-access-drppz\") pod \"payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7\" (UID: \"7f3ff7e0-2c72-45c4-bc9c-e58967921176\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7"
Nov 03 20:21:25.014627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:25.003072    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3ff7e0-2c72-45c4-bc9c-e58967921176-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7\" (UID: \"7f3ff7e0-2c72-45c4-bc9c-e58967921176\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7"
Nov 03 20:21:25.037620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:25.036244    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3ff7e0-2c72-45c4-bc9c-e58967921176-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7\" (UID: \"7f3ff7e0-2c72-45c4-bc9c-e58967921176\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7"
Nov 03 20:21:25.063153 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:25.062118    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drppz\" (UniqueName: \"kubernetes.io/projected/7f3ff7e0-2c72-45c4-bc9c-e58967921176-kube-api-access-drppz\") pod \"payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7\" (UID: \"7f3ff7e0-2c72-45c4-bc9c-e58967921176\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7"
Nov 03 20:21:25.180675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:25.171944    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7"
Nov 03 20:21:25.609664 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:25.602977    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7"]
Nov 03 20:21:25.883546 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:25.883467    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7" event={"ID":"7f3ff7e0-2c72-45c4-bc9c-e58967921176","Type":"ContainerStarted","Data":"bb7678f5c21d83e3680a695c598bf0c51f066dabc7d98018ba44a440d4f16e55"}
Nov 03 20:21:26.020004 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:26.019968    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1e67982-68de-4503-af48-8cb554c6679e" path="/var/lib/kubelet/pods/f1e67982-68de-4503-af48-8cb554c6679e/volumes"
Nov 03 20:21:26.580790 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:26.579627    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s" podUID="3db876db-2b38-4d10-88ab-4302077c072b" containerName="payload-pre-processing" containerID="cri-o://b35b660c15c57ec49036e421eb72b1a88adb2a20f2c8660386e81d0efc68a97e" gracePeriod=25
Nov 03 20:21:26.938855 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:26.936298    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7" event={"ID":"7f3ff7e0-2c72-45c4-bc9c-e58967921176","Type":"ContainerStarted","Data":"9ab9d6fbc35d54ea08a27af5075676edbf29e4f2e452fcbcba97cf3656d78548"}
Nov 03 20:21:26.938855 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:26.937773    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7"
Nov 03 20:21:26.978634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:26.976237    4421 generic.go:356] "Generic (PLEG): container finished" podID="3db876db-2b38-4d10-88ab-4302077c072b" containerID="b35b660c15c57ec49036e421eb72b1a88adb2a20f2c8660386e81d0efc68a97e" exitCode=0
Nov 03 20:21:26.978634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:26.976319    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s" event={"ID":"3db876db-2b38-4d10-88ab-4302077c072b","Type":"ContainerDied","Data":"b35b660c15c57ec49036e421eb72b1a88adb2a20f2c8660386e81d0efc68a97e"}
Nov 03 20:21:26.979399 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:26.979055    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7" podStartSLOduration=2.979036866 podStartE2EDuration="2.979036866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:26.976745417 +0000 UTC m=+1217.109873881" watchObservedRunningTime="2026-11-03 20:21:26.979036866 +0000 UTC m=+1217.112165328"
Nov 03 20:21:27.577804 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:27.577722    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s"
Nov 03 20:21:27.608230 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:27.608194    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3db876db-2b38-4d10-88ab-4302077c072b-kube-api-access-8hzqd\" (UniqueName: \"kubernetes.io/projected/3db876db-2b38-4d10-88ab-4302077c072b-kube-api-access-8hzqd\") pod \"3db876db-2b38-4d10-88ab-4302077c072b\" (UID: \"3db876db-2b38-4d10-88ab-4302077c072b\") "
Nov 03 20:21:27.608382 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:27.608308    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3db876db-2b38-4d10-88ab-4302077c072b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db876db-2b38-4d10-88ab-4302077c072b-plugins-config-volume\") pod \"3db876db-2b38-4d10-88ab-4302077c072b\" (UID: \"3db876db-2b38-4d10-88ab-4302077c072b\") "
Nov 03 20:21:27.609236 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:27.609207    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3db876db-2b38-4d10-88ab-4302077c072b-plugins-config-volume" pod "3db876db-2b38-4d10-88ab-4302077c072b" (UID: "3db876db-2b38-4d10-88ab-4302077c072b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:27.610088 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:27.610052    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db876db-2b38-4d10-88ab-4302077c072b-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:27.632759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:27.632706    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3db876db-2b38-4d10-88ab-4302077c072b-kube-api-access-8hzqd" pod "3db876db-2b38-4d10-88ab-4302077c072b" (UID: "3db876db-2b38-4d10-88ab-4302077c072b"). InnerVolumeSpecName "kube-api-access-8hzqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:27.711797 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:27.711766    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hzqd\" (UniqueName: \"kubernetes.io/projected/3db876db-2b38-4d10-88ab-4302077c072b-kube-api-access-8hzqd\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:27.991966 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:27.991912    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"]
Nov 03 20:21:27.993499 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:27.993473    4421 cpu_manager.go:414] "containerMap: removing container" podUID="3db876db-2b38-4d10-88ab-4302077c072b" containerName="payload-pre-processing"
Nov 03 20:21:27.993614 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:27.993500    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3db876db-2b38-4d10-88ab-4302077c072b" containerName="payload-pre-processing"
Nov 03 20:21:27.993688 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:27.993656    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="3db876db-2b38-4d10-88ab-4302077c072b" containerName="payload-pre-processing"
Nov 03 20:21:27.994493 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:27.994018    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s"
Nov 03 20:21:28.028502 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.027613    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.036852 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.036397    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2eee8c-openshift-default-dockercfg-q5wm8\""
Nov 03 20:21:28.063909 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.063874    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s" event={"ID":"3db876db-2b38-4d10-88ab-4302077c072b","Type":"ContainerDied","Data":"1d3dd6027f993d22cdbd8e778c7c5b3ed08c15c0285f807b28e50f6abdc62248"}
Nov 03 20:21:28.064051 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.063918    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"]
Nov 03 20:21:28.064051 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.063954    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s"]
Nov 03 20:21:28.064051 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.063977    4421 scope.go:122] "RemoveContainer" containerID="b35b660c15c57ec49036e421eb72b1a88adb2a20f2c8660386e81d0efc68a97e"
Nov 03 20:21:28.067734 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.067705    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d6789f-c97f6555f-bxg6s"]
Nov 03 20:21:28.166651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.134656    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-envoy\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.166651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.134722    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-data\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.166651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.134774    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-podinfo\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.166651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.134813    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-token\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.166651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.134868    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-workload-certs\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.166651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.135015    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5dq5\" (UniqueName: \"kubernetes.io/projected/f9ec3c59-274d-4af5-a710-751cf5367d3c-kube-api-access-w5dq5\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.166651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.135055    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-workload-socket\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.166651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.135126    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-credential-socket\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.166651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.135341    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ec3c59-274d-4af5-a710-751cf5367d3c-istiod-ca-cert\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.251688 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.250270    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ec3c59-274d-4af5-a710-751cf5367d3c-istiod-ca-cert\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.251688 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.250379    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-envoy\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.251688 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.250428    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-data\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.251688 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.250484    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-podinfo\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.251688 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.250520    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-token\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.251688 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.250562    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-workload-certs\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.251688 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.250701    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5dq5\" (UniqueName: \"kubernetes.io/projected/f9ec3c59-274d-4af5-a710-751cf5367d3c-kube-api-access-w5dq5\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.251688 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.250751    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-workload-socket\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.421684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.372527    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ec3c59-274d-4af5-a710-751cf5367d3c-istiod-ca-cert\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.421684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.374266    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-envoy\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.421684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.376438    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-workload-certs\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.421684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.380153    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-data\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.421684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.385079    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-credential-socket\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.421684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.385517    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-credential-socket\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.421684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.385731    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-workload-socket\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.465794 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.465701    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-token\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.533113 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.526142    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-podinfo\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.533113 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.526329    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5dq5\" (UniqueName: \"kubernetes.io/projected/f9ec3c59-274d-4af5-a710-751cf5367d3c-kube-api-access-w5dq5\") pod \"e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") " pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:28.676253 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:28.676209    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:29.053390 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:21:29.053340    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ec3c59_274d_4af5_a710_751cf5367d3c.slice/crio-fdc9bae017a96705e406195cc9c2fe377b15f6e23533a61ffa9c04a518a3dfd2 WatchSource:0}: Error finding container fdc9bae017a96705e406195cc9c2fe377b15f6e23533a61ffa9c04a518a3dfd2: Status 404 returned error can't find the container with id fdc9bae017a96705e406195cc9c2fe377b15f6e23533a61ffa9c04a518a3dfd2
Nov 03 20:21:29.054972 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:29.054945    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"]
Nov 03 20:21:29.056657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:29.056610    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:21:29.056759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:29.056702    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:21:29.056830 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:29.056767    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:21:29.976361 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:29.975862    4421 patch_prober.go:29] interesting pod/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": read tcp 10.134.0.2:37594->10.134.0.115:15021: read: connection reset by peer" start-of-body=
Nov 03 20:21:29.976361 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:29.975921    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh" podUID="10a12293-4a77-4829-93da-7e1909f3853f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": read tcp 10.134.0.2:37594->10.134.0.115:15021: read: connection reset by peer"
Nov 03 20:21:30.017191 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:30.017123    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3db876db-2b38-4d10-88ab-4302077c072b" path="/var/lib/kubelet/pods/3db876db-2b38-4d10-88ab-4302077c072b/volumes"
Nov 03 20:21:30.053677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:30.053644    4421 generic.go:356] "Generic (PLEG): container finished" podID="6ef60fe1-39dc-4f56-9114-d7d4e79a6b41" containerID="84c0fd0ca99d70a0ffe4ce66f9ba2d6bb05f0621b8d0e39cb980cf79fd521cc7" exitCode=0
Nov 03 20:21:30.054080 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:30.053716    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7" event={"ID":"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41","Type":"ContainerDied","Data":"84c0fd0ca99d70a0ffe4ce66f9ba2d6bb05f0621b8d0e39cb980cf79fd521cc7"}
Nov 03 20:21:30.054314 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:30.054297    4421 scope.go:122] "RemoveContainer" containerID="84c0fd0ca99d70a0ffe4ce66f9ba2d6bb05f0621b8d0e39cb980cf79fd521cc7"
Nov 03 20:21:30.055614 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:30.055575    4421 generic.go:356] "Generic (PLEG): container finished" podID="10a12293-4a77-4829-93da-7e1909f3853f" containerID="c37b2c431f861865fb9b00217e1225156c455c23ff225fa7331b785bb7cbcaf1" exitCode=0
Nov 03 20:21:30.055715 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:30.055638    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh" event={"ID":"10a12293-4a77-4829-93da-7e1909f3853f","Type":"ContainerDied","Data":"c37b2c431f861865fb9b00217e1225156c455c23ff225fa7331b785bb7cbcaf1"}
Nov 03 20:21:30.056027 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:30.056012    4421 scope.go:122] "RemoveContainer" containerID="c37b2c431f861865fb9b00217e1225156c455c23ff225fa7331b785bb7cbcaf1"
Nov 03 20:21:30.056274 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:30.056243    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:21:30.056349 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:30.056334    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:21:30.056404 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:30.056380    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:21:30.057413 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:30.057386    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:21:30.057488 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:30.057451    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:21:30.057528 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:30.057482    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq" event={"ID":"f9ec3c59-274d-4af5-a710-751cf5367d3c","Type":"ContainerStarted","Data":"b4cd17d481610745a638bc486c86fee13831ff6e632d37b3f23195c9e777f271"}
Nov 03 20:21:30.057565 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:30.057534    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq" event={"ID":"f9ec3c59-274d-4af5-a710-751cf5367d3c","Type":"ContainerStarted","Data":"fdc9bae017a96705e406195cc9c2fe377b15f6e23533a61ffa9c04a518a3dfd2"}
Nov 03 20:21:30.057565 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:30.057491    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:21:30.181351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:30.181284    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq" podStartSLOduration=3.181271999 podStartE2EDuration="3.181271999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:30.180660847 +0000 UTC m=+1220.313789311" watchObservedRunningTime="2026-11-03 20:21:30.181271999 +0000 UTC m=+1220.314400457"
Nov 03 20:21:30.680623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:30.676558    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:30.692619 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:30.690572    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:30.898676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:30.898644    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc"
Nov 03 20:21:31.197260 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:31.197226    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7" event={"ID":"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41","Type":"ContainerStarted","Data":"aa911352aae025c3c15423db9a6638e5274b047013673f9dc40cb5d74b02c856"}
Nov 03 20:21:31.221714 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:31.217412    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:31.221714 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:31.217459    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh" event={"ID":"10a12293-4a77-4829-93da-7e1909f3853f","Type":"ContainerStarted","Data":"a9c524e56122a534d3b85a6391912e7e74d6bdf7af3c6b0a6fae7f752170c8ed"}
Nov 03 20:21:31.221714 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:31.217526    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:21:31.290619 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:31.280816    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:21:31.290619 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:31.280942    4421 patch_prober.go:29] interesting pod/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 20:21:31.290619 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:31.280987    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7" podUID="6ef60fe1-39dc-4f56-9114-d7d4e79a6b41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 20:21:31.870622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:31.864961    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:21:31.870622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:31.865076    4421 patch_prober.go:29] interesting pod/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 03 20:21:31.870622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:31.865117    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh" podUID="10a12293-4a77-4829-93da-7e1909f3853f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 03 20:21:32.243621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:32.243115    4421 patch_prober.go:29] interesting pod/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 20:21:32.243621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:32.243166    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7" podUID="6ef60fe1-39dc-4f56-9114-d7d4e79a6b41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 20:21:32.865624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:32.865550    4421 patch_prober.go:29] interesting pod/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 03 20:21:32.865848 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:32.865646    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh" podUID="10a12293-4a77-4829-93da-7e1909f3853f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 03 20:21:33.240311 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:33.240260    4421 patch_prober.go:29] interesting pod/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 20:21:33.240523 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:33.240342    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7" podUID="6ef60fe1-39dc-4f56-9114-d7d4e79a6b41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 20:21:33.928791 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:33.928465    4421 patch_prober.go:29] interesting pod/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 03 20:21:33.928791 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:33.928523    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh" podUID="10a12293-4a77-4829-93da-7e1909f3853f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 03 20:21:34.327621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:34.320552    4421 patch_prober.go:29] interesting pod/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 20:21:34.327621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:34.320658    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7" podUID="6ef60fe1-39dc-4f56-9114-d7d4e79a6b41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 20:21:34.868011 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:34.867974    4421 patch_prober.go:29] interesting pod/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 03 20:21:34.868294 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:34.868262    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh" podUID="10a12293-4a77-4829-93da-7e1909f3853f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 03 20:21:35.194548 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:35.194498    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:21:35.238545 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:35.238458    4421 patch_prober.go:29] interesting pod/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 20:21:35.238808 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:35.238510    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7" podUID="6ef60fe1-39dc-4f56-9114-d7d4e79a6b41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 20:21:35.894624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:35.892497    4421 patch_prober.go:29] interesting pod/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 03 20:21:35.894624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:35.892557    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh" podUID="10a12293-4a77-4829-93da-7e1909f3853f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 03 20:21:36.238410 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:36.238377    4421 patch_prober.go:29] interesting pod/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 20:21:36.238844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:36.238425    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7" podUID="6ef60fe1-39dc-4f56-9114-d7d4e79a6b41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 20:21:36.872238 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:36.871777    4421 patch_prober.go:29] interesting pod/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 03 20:21:36.872238 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:36.871837    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh" podUID="10a12293-4a77-4829-93da-7e1909f3853f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 03 20:21:37.244278 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:37.244245    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:21:37.245344 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:37.245317    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:21:37.870954 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:37.870922    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:21:37.871397 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:37.871380    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:21:37.872091 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:37.872063    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:21:39.046769 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:39.046742    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7"
Nov 03 20:21:45.829532 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:45.829493    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6"]
Nov 03 20:21:45.834381 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:45.834359    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6"
Nov 03 20:21:45.844452 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:45.844414    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-2eee8c-serving-cert\""
Nov 03 20:21:45.853329 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:45.851955    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6"]
Nov 03 20:21:45.999409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:45.999361    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d0951ba-32cd-4e85-842a-2b487b838320-metrics-certs\") pod \"maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6\" (UID: \"8d0951ba-32cd-4e85-842a-2b487b838320\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6"
Nov 03 20:21:45.999656 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:45.999635    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d0951ba-32cd-4e85-842a-2b487b838320-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6\" (UID: \"8d0951ba-32cd-4e85-842a-2b487b838320\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6"
Nov 03 20:21:45.999804 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:45.999786    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwtkr\" (UniqueName: \"kubernetes.io/projected/8d0951ba-32cd-4e85-842a-2b487b838320-kube-api-access-jwtkr\") pod \"maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6\" (UID: \"8d0951ba-32cd-4e85-842a-2b487b838320\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6"
Nov 03 20:21:45.999883 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:45.999851    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d0951ba-32cd-4e85-842a-2b487b838320-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6\" (UID: \"8d0951ba-32cd-4e85-842a-2b487b838320\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6"
Nov 03 20:21:46.105991 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:46.105885    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d0951ba-32cd-4e85-842a-2b487b838320-metrics-certs\") pod \"maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6\" (UID: \"8d0951ba-32cd-4e85-842a-2b487b838320\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6"
Nov 03 20:21:46.106402 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:46.106381    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d0951ba-32cd-4e85-842a-2b487b838320-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6\" (UID: \"8d0951ba-32cd-4e85-842a-2b487b838320\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6"
Nov 03 20:21:46.106671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:46.106649    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwtkr\" (UniqueName: \"kubernetes.io/projected/8d0951ba-32cd-4e85-842a-2b487b838320-kube-api-access-jwtkr\") pod \"maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6\" (UID: \"8d0951ba-32cd-4e85-842a-2b487b838320\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6"
Nov 03 20:21:46.106836 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:46.106820    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d0951ba-32cd-4e85-842a-2b487b838320-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6\" (UID: \"8d0951ba-32cd-4e85-842a-2b487b838320\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6"
Nov 03 20:21:46.107838 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:46.107815    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d0951ba-32cd-4e85-842a-2b487b838320-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6\" (UID: \"8d0951ba-32cd-4e85-842a-2b487b838320\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6"
Nov 03 20:21:46.138659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:46.138574    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d0951ba-32cd-4e85-842a-2b487b838320-metrics-certs\") pod \"maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6\" (UID: \"8d0951ba-32cd-4e85-842a-2b487b838320\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6"
Nov 03 20:21:46.142505 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:46.142477    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d0951ba-32cd-4e85-842a-2b487b838320-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6\" (UID: \"8d0951ba-32cd-4e85-842a-2b487b838320\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6"
Nov 03 20:21:46.155705 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:46.155668    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwtkr\" (UniqueName: \"kubernetes.io/projected/8d0951ba-32cd-4e85-842a-2b487b838320-kube-api-access-jwtkr\") pod \"maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6\" (UID: \"8d0951ba-32cd-4e85-842a-2b487b838320\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6"
Nov 03 20:21:46.157355 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:46.157336    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6"
Nov 03 20:21:46.541326 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:46.528721    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6"]
Nov 03 20:21:47.134632 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:47.132507    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6" event={"ID":"8d0951ba-32cd-4e85-842a-2b487b838320","Type":"ContainerStarted","Data":"5b7b564c9367244c17fcb35818f30129d3c3ab872aa7a8650cc279f1007e51d6"}
Nov 03 20:21:48.141223 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:48.141182    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6" event={"ID":"8d0951ba-32cd-4e85-842a-2b487b838320","Type":"ContainerStarted","Data":"3185ddfbe8b11ceb64cb1f42b3892f4d5445206c70ed6850fc50336fe87854aa"}
Nov 03 20:21:48.141749 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:48.141253    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6"
Nov 03 20:21:48.162799 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:48.162747    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6" podStartSLOduration=2.500028818 podStartE2EDuration="3.16273266s" totalImagesPullingTime="662.703842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:48.160168497 +0000 UTC m=+1238.293296960" watchObservedRunningTime="2026-11-03 20:21:48.16273266 +0000 UTC m=+1238.295861122"
Nov 03 20:21:48.953166 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:48.953130    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82"]
Nov 03 20:21:49.638467 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:21:49.638422    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:21:53.814274 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:53.814239    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85"]
Nov 03 20:21:53.814795 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:53.814488    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85" podUID="5906e3ff-fc91-4434-93b1-3cd8c77dbae3" containerName="maas-api" containerID="cri-o://05b01b4040b44364f00313b3cdf743ce43be987ba70c69acdb38a9f01abcb32e" gracePeriod=30
Nov 03 20:21:53.834611 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:53.834557    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85" podUID="5906e3ff-fc91-4434-93b1-3cd8c77dbae3" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.116:8443/health\": dial tcp 10.134.0.116:8443: connect: connection refused"
Nov 03 20:21:53.953810 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:53.953658    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82" podUID="f0f698c3-76a3-41f3-b9d0-af1ea58a170e" containerName="payload-pre-processing" containerID="cri-o://8ac6e95e328c23763fcc4612dbd8c4688ba56cbbc6c597afff86f2c68b611f85" gracePeriod=25
Nov 03 20:21:54.344662 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:54.342849    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6"
Nov 03 20:21:54.359054 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:54.359008    4421 generic.go:356] "Generic (PLEG): container finished" podID="5906e3ff-fc91-4434-93b1-3cd8c77dbae3" containerID="05b01b4040b44364f00313b3cdf743ce43be987ba70c69acdb38a9f01abcb32e" exitCode=0
Nov 03 20:21:54.359229 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:54.359208    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85" event={"ID":"5906e3ff-fc91-4434-93b1-3cd8c77dbae3","Type":"ContainerDied","Data":"05b01b4040b44364f00313b3cdf743ce43be987ba70c69acdb38a9f01abcb32e"}
Nov 03 20:21:54.366153 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:54.365083    4421 generic.go:356] "Generic (PLEG): container finished" podID="f0f698c3-76a3-41f3-b9d0-af1ea58a170e" containerID="8ac6e95e328c23763fcc4612dbd8c4688ba56cbbc6c597afff86f2c68b611f85" exitCode=0
Nov 03 20:21:54.366153 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:54.365192    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82" event={"ID":"f0f698c3-76a3-41f3-b9d0-af1ea58a170e","Type":"ContainerDied","Data":"8ac6e95e328c23763fcc4612dbd8c4688ba56cbbc6c597afff86f2c68b611f85"}
Nov 03 20:21:54.416621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:54.416579    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85"
Nov 03 20:21:54.513280 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:54.513257    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82"
Nov 03 20:21:54.636692 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:54.636490    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-metrics-certs\") pod \"5906e3ff-fc91-4434-93b1-3cd8c77dbae3\" (UID: \"5906e3ff-fc91-4434-93b1-3cd8c77dbae3\") "
Nov 03 20:21:54.636692 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:54.636666    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-kube-api-access-bcp75\" (UniqueName: \"kubernetes.io/projected/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-kube-api-access-bcp75\") pod \"5906e3ff-fc91-4434-93b1-3cd8c77dbae3\" (UID: \"5906e3ff-fc91-4434-93b1-3cd8c77dbae3\") "
Nov 03 20:21:54.636929 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:54.636742    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0f698c3-76a3-41f3-b9d0-af1ea58a170e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f698c3-76a3-41f3-b9d0-af1ea58a170e-plugins-config-volume\") pod \"f0f698c3-76a3-41f3-b9d0-af1ea58a170e\" (UID: \"f0f698c3-76a3-41f3-b9d0-af1ea58a170e\") "
Nov 03 20:21:54.636929 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:54.636818    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-maas-api-tls\") pod \"5906e3ff-fc91-4434-93b1-3cd8c77dbae3\" (UID: \"5906e3ff-fc91-4434-93b1-3cd8c77dbae3\") "
Nov 03 20:21:54.636929 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:54.636887    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f698c3-76a3-41f3-b9d0-af1ea58a170e-kube-api-access-gcd82\" (UniqueName: \"kubernetes.io/projected/f0f698c3-76a3-41f3-b9d0-af1ea58a170e-kube-api-access-gcd82\") pod \"f0f698c3-76a3-41f3-b9d0-af1ea58a170e\" (UID: \"f0f698c3-76a3-41f3-b9d0-af1ea58a170e\") "
Nov 03 20:21:54.637079 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:54.637018    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-openshift-service-ca\") pod \"5906e3ff-fc91-4434-93b1-3cd8c77dbae3\" (UID: \"5906e3ff-fc91-4434-93b1-3cd8c77dbae3\") "
Nov 03 20:21:54.652650 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:54.652588    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f698c3-76a3-41f3-b9d0-af1ea58a170e-plugins-config-volume" pod "f0f698c3-76a3-41f3-b9d0-af1ea58a170e" (UID: "f0f698c3-76a3-41f3-b9d0-af1ea58a170e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:54.659657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:54.654164    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-openshift-service-ca" pod "5906e3ff-fc91-4434-93b1-3cd8c77dbae3" (UID: "5906e3ff-fc91-4434-93b1-3cd8c77dbae3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:54.674282 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:54.674201    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-maas-api-tls" pod "5906e3ff-fc91-4434-93b1-3cd8c77dbae3" (UID: "5906e3ff-fc91-4434-93b1-3cd8c77dbae3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:21:54.678382 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:54.678348    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-metrics-certs" pod "5906e3ff-fc91-4434-93b1-3cd8c77dbae3" (UID: "5906e3ff-fc91-4434-93b1-3cd8c77dbae3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:21:54.678939 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:54.678906    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f698c3-76a3-41f3-b9d0-af1ea58a170e-kube-api-access-gcd82" pod "f0f698c3-76a3-41f3-b9d0-af1ea58a170e" (UID: "f0f698c3-76a3-41f3-b9d0-af1ea58a170e"). InnerVolumeSpecName "kube-api-access-gcd82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:54.679979 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:54.679953    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-kube-api-access-bcp75" pod "5906e3ff-fc91-4434-93b1-3cd8c77dbae3" (UID: "5906e3ff-fc91-4434-93b1-3cd8c77dbae3"). InnerVolumeSpecName "kube-api-access-bcp75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:54.739428 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:54.739388    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:54.739428 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:54.739420    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcp75\" (UniqueName: \"kubernetes.io/projected/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-kube-api-access-bcp75\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:54.739428 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:54.739435    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f698c3-76a3-41f3-b9d0-af1ea58a170e-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:54.739691 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:54.739448    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:54.739691 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:54.739461    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcd82\" (UniqueName: \"kubernetes.io/projected/f0f698c3-76a3-41f3-b9d0-af1ea58a170e-kube-api-access-gcd82\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:54.739691 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:54.739473    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5906e3ff-fc91-4434-93b1-3cd8c77dbae3-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:55.403584 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:55.403546    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82" event={"ID":"f0f698c3-76a3-41f3-b9d0-af1ea58a170e","Type":"ContainerDied","Data":"d8d0dbf3f066a683952b7a95ea563e10dae411e8c41156a279a594d1b00c7b3b"}
Nov 03 20:21:55.404221 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:55.404200    4421 scope.go:122] "RemoveContainer" containerID="8ac6e95e328c23763fcc4612dbd8c4688ba56cbbc6c597afff86f2c68b611f85"
Nov 03 20:21:55.404708 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:55.404690    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82"
Nov 03 20:21:55.411494 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:55.408112    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85" event={"ID":"5906e3ff-fc91-4434-93b1-3cd8c77dbae3","Type":"ContainerDied","Data":"7ac4334f6f15a5321f9665d9204c16c62a8c11b7f7161f30f9b4d01805b3441d"}
Nov 03 20:21:55.411494 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:55.408223    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85"
Nov 03 20:21:55.423018 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:55.422994    4421 scope.go:122] "RemoveContainer" containerID="05b01b4040b44364f00313b3cdf743ce43be987ba70c69acdb38a9f01abcb32e"
Nov 03 20:21:55.439715 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:55.439695    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82"]
Nov 03 20:21:55.443181 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:55.443160    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f2218a-55d586c86c-svk82"]
Nov 03 20:21:55.452197 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:55.452177    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85"]
Nov 03 20:21:55.456714 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:55.456695    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f2218a-867759db75-qgh85"]
Nov 03 20:21:56.020250 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:56.020215    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5906e3ff-fc91-4434-93b1-3cd8c77dbae3" path="/var/lib/kubelet/pods/5906e3ff-fc91-4434-93b1-3cd8c77dbae3/volumes"
Nov 03 20:21:56.020736 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:56.020718    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0f698c3-76a3-41f3-b9d0-af1ea58a170e" path="/var/lib/kubelet/pods/f0f698c3-76a3-41f3-b9d0-af1ea58a170e/volumes"
Nov 03 20:21:57.035817 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:57.035787    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"]
Nov 03 20:21:57.036343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:57.035998    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7" podUID="6ef60fe1-39dc-4f56-9114-d7d4e79a6b41" containerName="istio-proxy" containerID="cri-o://aa911352aae025c3c15423db9a6638e5274b047013673f9dc40cb5d74b02c856" gracePeriod=30
Nov 03 20:21:57.084883 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:57.084852    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg"]
Nov 03 20:21:57.085159 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:57.085131    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg" podUID="f9259c57-03ab-490b-bbde-f40be00f9ad4" containerName="payload-processing" containerID="cri-o://e529299b72cd908a5c4c6432521b49c59460c49640c37d0d281d3ef23595caf4" gracePeriod=30
Nov 03 20:21:57.094743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:57.094715    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj"]
Nov 03 20:21:57.094965 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:57.094940    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj" podUID="3ab7a061-3614-41d6-af5a-25dde57e6bbd" containerName="payload-pre-processing" containerID="cri-o://81676fba2a7e805e56718cf592a18c6037d76f11105b5353a157931c9ee73c23" gracePeriod=30
Nov 03 20:21:57.720167 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:57.720116    4421 generic.go:356] "Generic (PLEG): container finished" podID="f9259c57-03ab-490b-bbde-f40be00f9ad4" containerID="e529299b72cd908a5c4c6432521b49c59460c49640c37d0d281d3ef23595caf4" exitCode=0
Nov 03 20:21:57.720987 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:57.720339    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg" event={"ID":"f9259c57-03ab-490b-bbde-f40be00f9ad4","Type":"ContainerDied","Data":"e529299b72cd908a5c4c6432521b49c59460c49640c37d0d281d3ef23595caf4"}
Nov 03 20:21:57.723764 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:57.723697    4421 generic.go:356] "Generic (PLEG): container finished" podID="3ab7a061-3614-41d6-af5a-25dde57e6bbd" containerID="81676fba2a7e805e56718cf592a18c6037d76f11105b5353a157931c9ee73c23" exitCode=0
Nov 03 20:21:57.723872 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:57.723851    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj" event={"ID":"3ab7a061-3614-41d6-af5a-25dde57e6bbd","Type":"ContainerDied","Data":"81676fba2a7e805e56718cf592a18c6037d76f11105b5353a157931c9ee73c23"}
Nov 03 20:21:57.738069 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:57.738042    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg"
Nov 03 20:21:57.800285 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:57.798987    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9259c57-03ab-490b-bbde-f40be00f9ad4-kube-api-access-d4sw8\" (UniqueName: \"kubernetes.io/projected/f9259c57-03ab-490b-bbde-f40be00f9ad4-kube-api-access-d4sw8\") pod \"f9259c57-03ab-490b-bbde-f40be00f9ad4\" (UID: \"f9259c57-03ab-490b-bbde-f40be00f9ad4\") "
Nov 03 20:21:57.800285 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:57.799153    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9259c57-03ab-490b-bbde-f40be00f9ad4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9259c57-03ab-490b-bbde-f40be00f9ad4-plugins-config-volume\") pod \"f9259c57-03ab-490b-bbde-f40be00f9ad4\" (UID: \"f9259c57-03ab-490b-bbde-f40be00f9ad4\") "
Nov 03 20:21:57.801051 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:57.800734    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9259c57-03ab-490b-bbde-f40be00f9ad4-plugins-config-volume" pod "f9259c57-03ab-490b-bbde-f40be00f9ad4" (UID: "f9259c57-03ab-490b-bbde-f40be00f9ad4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:57.807464 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:57.807422    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9259c57-03ab-490b-bbde-f40be00f9ad4-kube-api-access-d4sw8" pod "f9259c57-03ab-490b-bbde-f40be00f9ad4" (UID: "f9259c57-03ab-490b-bbde-f40be00f9ad4"). InnerVolumeSpecName "kube-api-access-d4sw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:57.809324 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:57.809027    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj"
Nov 03 20:21:57.959497 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:57.956938    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ab7a061-3614-41d6-af5a-25dde57e6bbd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ab7a061-3614-41d6-af5a-25dde57e6bbd-plugins-config-volume\") pod \"3ab7a061-3614-41d6-af5a-25dde57e6bbd\" (UID: \"3ab7a061-3614-41d6-af5a-25dde57e6bbd\") "
Nov 03 20:21:57.976359 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:57.976322    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ab7a061-3614-41d6-af5a-25dde57e6bbd-kube-api-access-htstd\" (UniqueName: \"kubernetes.io/projected/3ab7a061-3614-41d6-af5a-25dde57e6bbd-kube-api-access-htstd\") pod \"3ab7a061-3614-41d6-af5a-25dde57e6bbd\" (UID: \"3ab7a061-3614-41d6-af5a-25dde57e6bbd\") "
Nov 03 20:21:57.995664 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:57.986470    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9259c57-03ab-490b-bbde-f40be00f9ad4-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:57.995917 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:57.995896    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4sw8\" (UniqueName: \"kubernetes.io/projected/f9259c57-03ab-490b-bbde-f40be00f9ad4-kube-api-access-d4sw8\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:58.005803 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:58.005765    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ab7a061-3614-41d6-af5a-25dde57e6bbd-kube-api-access-htstd" pod "3ab7a061-3614-41d6-af5a-25dde57e6bbd" (UID: "3ab7a061-3614-41d6-af5a-25dde57e6bbd"). InnerVolumeSpecName "kube-api-access-htstd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:58.013956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:58.009820    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ab7a061-3614-41d6-af5a-25dde57e6bbd-plugins-config-volume" pod "3ab7a061-3614-41d6-af5a-25dde57e6bbd" (UID: "3ab7a061-3614-41d6-af5a-25dde57e6bbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:58.097623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:58.097512    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htstd\" (UniqueName: \"kubernetes.io/projected/3ab7a061-3614-41d6-af5a-25dde57e6bbd-kube-api-access-htstd\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:58.098150 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:58.098130    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ab7a061-3614-41d6-af5a-25dde57e6bbd-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:58.732082 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:58.732055    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg"
Nov 03 20:21:58.732082 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:58.732076    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg" event={"ID":"f9259c57-03ab-490b-bbde-f40be00f9ad4","Type":"ContainerDied","Data":"3b5fcfb8aa894696f16bd5efd31289ee338306bb11340d16d8e2ede1eaa95461"}
Nov 03 20:21:58.732335 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:58.732131    4421 scope.go:122] "RemoveContainer" containerID="e529299b72cd908a5c4c6432521b49c59460c49640c37d0d281d3ef23595caf4"
Nov 03 20:21:58.733958 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:58.733940    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj"
Nov 03 20:21:58.734085 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:58.733975    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj" event={"ID":"3ab7a061-3614-41d6-af5a-25dde57e6bbd","Type":"ContainerDied","Data":"5eb6a3844e411a59557f38db0ba435c48d3b14bb5d8ff43b4d38b159fd40a79c"}
Nov 03 20:21:58.742839 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:58.742818    4421 scope.go:122] "RemoveContainer" containerID="81676fba2a7e805e56718cf592a18c6037d76f11105b5353a157931c9ee73c23"
Nov 03 20:21:58.755171 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:58.755148    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg"]
Nov 03 20:21:58.759898 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:58.759879    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1b7b5e-7fd77b586-lkpvg"]
Nov 03 20:21:58.776690 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:58.776666    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj"]
Nov 03 20:21:58.781978 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:58.781955    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1b7b5e-85c67c759-kh8lj"]
Nov 03 20:21:59.030392 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.030313    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5"]
Nov 03 20:21:59.031320 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.031295    4421 cpu_manager.go:414] "containerMap: removing container" podUID="3ab7a061-3614-41d6-af5a-25dde57e6bbd" containerName="payload-pre-processing"
Nov 03 20:21:59.031320 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.031317    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ab7a061-3614-41d6-af5a-25dde57e6bbd" containerName="payload-pre-processing"
Nov 03 20:21:59.031475 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.031330    4421 cpu_manager.go:414] "containerMap: removing container" podUID="f0f698c3-76a3-41f3-b9d0-af1ea58a170e" containerName="payload-pre-processing"
Nov 03 20:21:59.031475 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.031336    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f698c3-76a3-41f3-b9d0-af1ea58a170e" containerName="payload-pre-processing"
Nov 03 20:21:59.031475 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.031357    4421 cpu_manager.go:414] "containerMap: removing container" podUID="5906e3ff-fc91-4434-93b1-3cd8c77dbae3" containerName="maas-api"
Nov 03 20:21:59.031475 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.031361    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5906e3ff-fc91-4434-93b1-3cd8c77dbae3" containerName="maas-api"
Nov 03 20:21:59.031475 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.031378    4421 cpu_manager.go:414] "containerMap: removing container" podUID="f9259c57-03ab-490b-bbde-f40be00f9ad4" containerName="payload-processing"
Nov 03 20:21:59.031475 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.031383    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9259c57-03ab-490b-bbde-f40be00f9ad4" containerName="payload-processing"
Nov 03 20:21:59.031475 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.031473    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ab7a061-3614-41d6-af5a-25dde57e6bbd" containerName="payload-pre-processing"
Nov 03 20:21:59.031728 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.031484    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f698c3-76a3-41f3-b9d0-af1ea58a170e" containerName="payload-pre-processing"
Nov 03 20:21:59.031728 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.031492    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="5906e3ff-fc91-4434-93b1-3cd8c77dbae3" containerName="maas-api"
Nov 03 20:21:59.031728 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.031498    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9259c57-03ab-490b-bbde-f40be00f9ad4" containerName="payload-processing"
Nov 03 20:21:59.036307 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.036288    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5"
Nov 03 20:21:59.051705 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.051679    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5"]
Nov 03 20:21:59.060902 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.060870    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f"]
Nov 03 20:21:59.065941 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.065914    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f"
Nov 03 20:21:59.072017 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.071977    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-efba68-dockercfg-sjgmb\""
Nov 03 20:21:59.075712 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.075689    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f"]
Nov 03 20:21:59.112552 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.112528    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-efba68\""
Nov 03 20:21:59.118501 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.118476    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxmwb\" (UniqueName: \"kubernetes.io/projected/a4345985-0271-49fe-8cfa-8478d88209c7-kube-api-access-wxmwb\") pod \"payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f\" (UID: \"a4345985-0271-49fe-8cfa-8478d88209c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f"
Nov 03 20:21:59.118638 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.118507    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-892t9\" (UniqueName: \"kubernetes.io/projected/11a98991-daa5-43ef-ad1d-e378920a76e5-kube-api-access-892t9\") pod \"payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5\" (UID: \"11a98991-daa5-43ef-ad1d-e378920a76e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5"
Nov 03 20:21:59.118722 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.118694    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a98991-daa5-43ef-ad1d-e378920a76e5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5\" (UID: \"11a98991-daa5-43ef-ad1d-e378920a76e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5"
Nov 03 20:21:59.118792 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.118776    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4345985-0271-49fe-8cfa-8478d88209c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f\" (UID: \"a4345985-0271-49fe-8cfa-8478d88209c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f"
Nov 03 20:21:59.234630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.230682    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxmwb\" (UniqueName: \"kubernetes.io/projected/a4345985-0271-49fe-8cfa-8478d88209c7-kube-api-access-wxmwb\") pod \"payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f\" (UID: \"a4345985-0271-49fe-8cfa-8478d88209c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f"
Nov 03 20:21:59.234630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.230739    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-892t9\" (UniqueName: \"kubernetes.io/projected/11a98991-daa5-43ef-ad1d-e378920a76e5-kube-api-access-892t9\") pod \"payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5\" (UID: \"11a98991-daa5-43ef-ad1d-e378920a76e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5"
Nov 03 20:21:59.234630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.230843    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a98991-daa5-43ef-ad1d-e378920a76e5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5\" (UID: \"11a98991-daa5-43ef-ad1d-e378920a76e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5"
Nov 03 20:21:59.234630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.230909    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4345985-0271-49fe-8cfa-8478d88209c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f\" (UID: \"a4345985-0271-49fe-8cfa-8478d88209c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f"
Nov 03 20:21:59.234630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.232130    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4345985-0271-49fe-8cfa-8478d88209c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f\" (UID: \"a4345985-0271-49fe-8cfa-8478d88209c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f"
Nov 03 20:21:59.234630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.233417    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a98991-daa5-43ef-ad1d-e378920a76e5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5\" (UID: \"11a98991-daa5-43ef-ad1d-e378920a76e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5"
Nov 03 20:21:59.271634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.271342    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-892t9\" (UniqueName: \"kubernetes.io/projected/11a98991-daa5-43ef-ad1d-e378920a76e5-kube-api-access-892t9\") pod \"payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5\" (UID: \"11a98991-daa5-43ef-ad1d-e378920a76e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5"
Nov 03 20:21:59.300240 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.300015    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxmwb\" (UniqueName: \"kubernetes.io/projected/a4345985-0271-49fe-8cfa-8478d88209c7-kube-api-access-wxmwb\") pod \"payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f\" (UID: \"a4345985-0271-49fe-8cfa-8478d88209c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f"
Nov 03 20:21:59.382620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.381666    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5"
Nov 03 20:21:59.387186 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.386897    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f"
Nov 03 20:21:59.763999 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.763970    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f"]
Nov 03 20:21:59.764128 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:21:59.763975    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4345985_0271_49fe_8cfa_8478d88209c7.slice/crio-7127e5bdc77fa5c06f2767ce541f1de9e39d17f5790e7b1e4609726c2e25d1a2 WatchSource:0}: Error finding container 7127e5bdc77fa5c06f2767ce541f1de9e39d17f5790e7b1e4609726c2e25d1a2: Status 404 returned error can't find the container with id 7127e5bdc77fa5c06f2767ce541f1de9e39d17f5790e7b1e4609726c2e25d1a2
Nov 03 20:21:59.779280 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:21:59.778614    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5"]
Nov 03 20:21:59.782987 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:21:59.782964    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a98991_daa5_43ef_ad1d_e378920a76e5.slice/crio-41624f519acd2aac0c0ab52b2dd5651c8dfa7fb7a48cf2680c02f37ac04cc4f0 WatchSource:0}: Error finding container 41624f519acd2aac0c0ab52b2dd5651c8dfa7fb7a48cf2680c02f37ac04cc4f0: Status 404 returned error can't find the container with id 41624f519acd2aac0c0ab52b2dd5651c8dfa7fb7a48cf2680c02f37ac04cc4f0
Nov 03 20:22:00.019865 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:00.019831    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ab7a061-3614-41d6-af5a-25dde57e6bbd" path="/var/lib/kubelet/pods/3ab7a061-3614-41d6-af5a-25dde57e6bbd/volumes"
Nov 03 20:22:00.020399 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:00.020383    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9259c57-03ab-490b-bbde-f40be00f9ad4" path="/var/lib/kubelet/pods/f9259c57-03ab-490b-bbde-f40be00f9ad4/volumes"
Nov 03 20:22:00.814188 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:00.814146    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5" event={"ID":"11a98991-daa5-43ef-ad1d-e378920a76e5","Type":"ContainerStarted","Data":"6690e7e83863665717f8ee04a496da3b0f105a090e8cf973d7eb2f3e30a5456e"}
Nov 03 20:22:00.814807 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:00.814216    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5" event={"ID":"11a98991-daa5-43ef-ad1d-e378920a76e5","Type":"ContainerStarted","Data":"41624f519acd2aac0c0ab52b2dd5651c8dfa7fb7a48cf2680c02f37ac04cc4f0"}
Nov 03 20:22:00.814980 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:00.814964    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5"
Nov 03 20:22:00.817522 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:00.817479    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f" event={"ID":"a4345985-0271-49fe-8cfa-8478d88209c7","Type":"ContainerStarted","Data":"1efd2e948b8102edfa8619f77ae069474c955de4c8253c22ee30c8284463c4a1"}
Nov 03 20:22:00.817718 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:00.817538    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f" event={"ID":"a4345985-0271-49fe-8cfa-8478d88209c7","Type":"ContainerStarted","Data":"7127e5bdc77fa5c06f2767ce541f1de9e39d17f5790e7b1e4609726c2e25d1a2"}
Nov 03 20:22:00.817718 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:00.817563    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f"
Nov 03 20:22:00.837538 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:00.837487    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5" podStartSLOduration=1.8374732310000002 podStartE2EDuration="1.837473231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:00.832748393 +0000 UTC m=+1250.965876855" watchObservedRunningTime="2026-11-03 20:22:00.837473231 +0000 UTC m=+1250.970601727"
Nov 03 20:22:00.854554 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:00.854502    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f" podStartSLOduration=1.854487097 podStartE2EDuration="1.854487097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:00.850504018 +0000 UTC m=+1250.983632481" watchObservedRunningTime="2026-11-03 20:22:00.854487097 +0000 UTC m=+1250.987615558"
Nov 03 20:22:02.330371 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.330298    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:22:02.449197 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.449163    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-token\" (UniqueName: \"kubernetes.io/projected/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-token\") pod \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") "
Nov 03 20:22:02.449370 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.449264    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istiod-ca-cert\") pod \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") "
Nov 03 20:22:02.449370 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.449344    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-envoy\") pod \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") "
Nov 03 20:22:02.449463 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.449394    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-data\") pod \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") "
Nov 03 20:22:02.449463 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.449426    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-podinfo\") pod \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") "
Nov 03 20:22:02.449463 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.449454    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-workload-socket\") pod \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") "
Nov 03 20:22:02.449636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.449500    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-credential-socket\") pod \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") "
Nov 03 20:22:02.449636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.449561    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-workload-certs\") pod \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") "
Nov 03 20:22:02.449744 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.449640    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-kube-api-access-9pbbq\" (UniqueName: \"kubernetes.io/projected/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-kube-api-access-9pbbq\") pod \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\" (UID: \"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41\") "
Nov 03 20:22:02.449813 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.449741    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-data" pod "6ef60fe1-39dc-4f56-9114-d7d4e79a6b41" (UID: "6ef60fe1-39dc-4f56-9114-d7d4e79a6b41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:02.449813 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.449767    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-workload-socket" pod "6ef60fe1-39dc-4f56-9114-d7d4e79a6b41" (UID: "6ef60fe1-39dc-4f56-9114-d7d4e79a6b41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:02.449813 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.449779    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istiod-ca-cert" pod "6ef60fe1-39dc-4f56-9114-d7d4e79a6b41" (UID: "6ef60fe1-39dc-4f56-9114-d7d4e79a6b41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:22:02.450002 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.449969    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-credential-socket" pod "6ef60fe1-39dc-4f56-9114-d7d4e79a6b41" (UID: "6ef60fe1-39dc-4f56-9114-d7d4e79a6b41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:02.450002 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.449985    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-workload-certs" pod "6ef60fe1-39dc-4f56-9114-d7d4e79a6b41" (UID: "6ef60fe1-39dc-4f56-9114-d7d4e79a6b41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:02.450453 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.450385    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:02.450453 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.450404    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:02.450453 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.450413    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:02.450453 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.450425    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:02.450453 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.450434    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:02.452338 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.452308    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-kube-api-access-9pbbq" pod "6ef60fe1-39dc-4f56-9114-d7d4e79a6b41" (UID: "6ef60fe1-39dc-4f56-9114-d7d4e79a6b41"). InnerVolumeSpecName "kube-api-access-9pbbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:02.452338 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.452316    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-envoy" pod "6ef60fe1-39dc-4f56-9114-d7d4e79a6b41" (UID: "6ef60fe1-39dc-4f56-9114-d7d4e79a6b41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:02.452496 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.452432    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-token" pod "6ef60fe1-39dc-4f56-9114-d7d4e79a6b41" (UID: "6ef60fe1-39dc-4f56-9114-d7d4e79a6b41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:02.452496 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.452465    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-podinfo" pod "6ef60fe1-39dc-4f56-9114-d7d4e79a6b41" (UID: "6ef60fe1-39dc-4f56-9114-d7d4e79a6b41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:22:02.552170 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.552140    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:02.552170 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.552165    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:02.552170 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.552174    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:02.552406 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.552185    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pbbq\" (UniqueName: \"kubernetes.io/projected/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41-kube-api-access-9pbbq\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:02.925238 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.925201    4421 generic.go:356] "Generic (PLEG): container finished" podID="6ef60fe1-39dc-4f56-9114-d7d4e79a6b41" containerID="aa911352aae025c3c15423db9a6638e5274b047013673f9dc40cb5d74b02c856" exitCode=0
Nov 03 20:22:02.925431 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.925266    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"
Nov 03 20:22:02.925431 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.925288    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7" event={"ID":"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41","Type":"ContainerDied","Data":"aa911352aae025c3c15423db9a6638e5274b047013673f9dc40cb5d74b02c856"}
Nov 03 20:22:02.925431 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.925344    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7" event={"ID":"6ef60fe1-39dc-4f56-9114-d7d4e79a6b41","Type":"ContainerDied","Data":"6e3967d52ffbc3366b3c2a50bd127d6213d934511b611eae40e5ee5d12131590"}
Nov 03 20:22:02.925431 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.925375    4421 scope.go:122] "RemoveContainer" containerID="aa911352aae025c3c15423db9a6638e5274b047013673f9dc40cb5d74b02c856"
Nov 03 20:22:02.938492 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.938472    4421 scope.go:122] "RemoveContainer" containerID="84c0fd0ca99d70a0ffe4ce66f9ba2d6bb05f0621b8d0e39cb980cf79fd521cc7"
Nov 03 20:22:02.951108 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.951086    4421 scope.go:122] "RemoveContainer" containerID="aa911352aae025c3c15423db9a6638e5274b047013673f9dc40cb5d74b02c856"
Nov 03 20:22:02.951447 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:22:02.951423    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa911352aae025c3c15423db9a6638e5274b047013673f9dc40cb5d74b02c856\": container with ID starting with aa911352aae025c3c15423db9a6638e5274b047013673f9dc40cb5d74b02c856 not found: ID does not exist" containerID="aa911352aae025c3c15423db9a6638e5274b047013673f9dc40cb5d74b02c856"
Nov 03 20:22:02.951541 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.951453    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa911352aae025c3c15423db9a6638e5274b047013673f9dc40cb5d74b02c856"} err="failed to get container status \"aa911352aae025c3c15423db9a6638e5274b047013673f9dc40cb5d74b02c856\": rpc error: code = NotFound desc = could not find container \"aa911352aae025c3c15423db9a6638e5274b047013673f9dc40cb5d74b02c856\": container with ID starting with aa911352aae025c3c15423db9a6638e5274b047013673f9dc40cb5d74b02c856 not found: ID does not exist"
Nov 03 20:22:02.951541 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.951469    4421 scope.go:122] "RemoveContainer" containerID="84c0fd0ca99d70a0ffe4ce66f9ba2d6bb05f0621b8d0e39cb980cf79fd521cc7"
Nov 03 20:22:02.951805 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:22:02.951776    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84c0fd0ca99d70a0ffe4ce66f9ba2d6bb05f0621b8d0e39cb980cf79fd521cc7\": container with ID starting with 84c0fd0ca99d70a0ffe4ce66f9ba2d6bb05f0621b8d0e39cb980cf79fd521cc7 not found: ID does not exist" containerID="84c0fd0ca99d70a0ffe4ce66f9ba2d6bb05f0621b8d0e39cb980cf79fd521cc7"
Nov 03 20:22:02.951864 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.951816    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84c0fd0ca99d70a0ffe4ce66f9ba2d6bb05f0621b8d0e39cb980cf79fd521cc7"} err="failed to get container status \"84c0fd0ca99d70a0ffe4ce66f9ba2d6bb05f0621b8d0e39cb980cf79fd521cc7\": rpc error: code = NotFound desc = could not find container \"84c0fd0ca99d70a0ffe4ce66f9ba2d6bb05f0621b8d0e39cb980cf79fd521cc7\": container with ID starting with 84c0fd0ca99d70a0ffe4ce66f9ba2d6bb05f0621b8d0e39cb980cf79fd521cc7 not found: ID does not exist"
Nov 03 20:22:02.954740 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.954700    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"]
Nov 03 20:22:02.958341 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:02.958321    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f2218a-openshift-default-5dcc9db77c-kmld7"]
Nov 03 20:22:03.486701 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:03.486670    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"]
Nov 03 20:22:03.489499 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:03.489437    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh" podUID="9ffda400-707f-48d5-9e1f-7f76caf6cd58" containerName="istio-proxy" containerID="cri-o://90e158f7360be80806deefad728af0023158e6310164b18ffd144e8c090481c7" gracePeriod=30
Nov 03 20:22:04.024257 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:04.024221    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ef60fe1-39dc-4f56-9114-d7d4e79a6b41" path="/var/lib/kubelet/pods/6ef60fe1-39dc-4f56-9114-d7d4e79a6b41/volumes"
Nov 03 20:22:08.525326 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:08.525294    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"]
Nov 03 20:22:08.526172 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:08.525580    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn" podUID="249f4a98-95d2-4a3c-9cfc-a6f00e214815" containerName="istio-proxy" containerID="cri-o://c81421f21771c03879aabdfdc40c47e1bd18fd88744c2c3d822738cd0ace0e82" gracePeriod=30
Nov 03 20:22:08.987869 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:08.987827    4421 generic.go:356] "Generic (PLEG): container finished" podID="9ffda400-707f-48d5-9e1f-7f76caf6cd58" containerID="90e158f7360be80806deefad728af0023158e6310164b18ffd144e8c090481c7" exitCode=0
Nov 03 20:22:08.988061 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:08.987964    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh" event={"ID":"9ffda400-707f-48d5-9e1f-7f76caf6cd58","Type":"ContainerDied","Data":"90e158f7360be80806deefad728af0023158e6310164b18ffd144e8c090481c7"}
Nov 03 20:22:08.988061 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:08.988027    4421 scope.go:122] "RemoveContainer" containerID="d488539e315ffe818f2c07db4c61c67d6a4b3d20960e949dc1949477aae66ca8"
Nov 03 20:22:09.048569 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.048539    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:22:09.152402 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.152309    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-credential-socket\") pod \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") "
Nov 03 20:22:09.152402 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.152373    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-podinfo\") pod \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") "
Nov 03 20:22:09.152664 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.152406    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istiod-ca-cert\") pod \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") "
Nov 03 20:22:09.152664 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.152441    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-workload-socket\") pod \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") "
Nov 03 20:22:09.152664 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.152480    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ffda400-707f-48d5-9e1f-7f76caf6cd58-kube-api-access-tcf8b\" (UniqueName: \"kubernetes.io/projected/9ffda400-707f-48d5-9e1f-7f76caf6cd58-kube-api-access-tcf8b\") pod \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") "
Nov 03 20:22:09.152664 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.152528    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-token\" (UniqueName: \"kubernetes.io/projected/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-token\") pod \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") "
Nov 03 20:22:09.152664 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.152626    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-credential-socket" pod "9ffda400-707f-48d5-9e1f-7f76caf6cd58" (UID: "9ffda400-707f-48d5-9e1f-7f76caf6cd58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:09.152937 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.152699    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-envoy\") pod \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") "
Nov 03 20:22:09.152937 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.152784    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-workload-certs\") pod \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") "
Nov 03 20:22:09.152937 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.152856    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-data\") pod \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\" (UID: \"9ffda400-707f-48d5-9e1f-7f76caf6cd58\") "
Nov 03 20:22:09.153121 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.153092    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istiod-ca-cert" pod "9ffda400-707f-48d5-9e1f-7f76caf6cd58" (UID: "9ffda400-707f-48d5-9e1f-7f76caf6cd58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:22:09.153439 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.153368    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-workload-socket" pod "9ffda400-707f-48d5-9e1f-7f76caf6cd58" (UID: "9ffda400-707f-48d5-9e1f-7f76caf6cd58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:09.153546 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.153500    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-workload-certs" pod "9ffda400-707f-48d5-9e1f-7f76caf6cd58" (UID: "9ffda400-707f-48d5-9e1f-7f76caf6cd58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:09.153672 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.153647    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-data" pod "9ffda400-707f-48d5-9e1f-7f76caf6cd58" (UID: "9ffda400-707f-48d5-9e1f-7f76caf6cd58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:09.153791 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.153671    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:09.153791 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.153694    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:09.153791 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.153709    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:09.155137 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.155108    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-podinfo" pod "9ffda400-707f-48d5-9e1f-7f76caf6cd58" (UID: "9ffda400-707f-48d5-9e1f-7f76caf6cd58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:22:09.155670 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.155641    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ffda400-707f-48d5-9e1f-7f76caf6cd58-kube-api-access-tcf8b" pod "9ffda400-707f-48d5-9e1f-7f76caf6cd58" (UID: "9ffda400-707f-48d5-9e1f-7f76caf6cd58"). InnerVolumeSpecName "kube-api-access-tcf8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:09.155756 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.155681    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-envoy" pod "9ffda400-707f-48d5-9e1f-7f76caf6cd58" (UID: "9ffda400-707f-48d5-9e1f-7f76caf6cd58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:09.155797 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.155758    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-token" pod "9ffda400-707f-48d5-9e1f-7f76caf6cd58" (UID: "9ffda400-707f-48d5-9e1f-7f76caf6cd58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:09.255062 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.255025    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:09.255062 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.255054    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:09.255062 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.255064    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:09.255289 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.255073    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcf8b\" (UniqueName: \"kubernetes.io/projected/9ffda400-707f-48d5-9e1f-7f76caf6cd58-kube-api-access-tcf8b\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:09.255289 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.255083    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:09.255289 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.255092    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ffda400-707f-48d5-9e1f-7f76caf6cd58-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:09.997040 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.997006    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"
Nov 03 20:22:09.997510 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.997006    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh" event={"ID":"9ffda400-707f-48d5-9e1f-7f76caf6cd58","Type":"ContainerDied","Data":"635e4f85666fcda5d78159e89395f69e8b71acd84e755366131e8b9aff7c1df4"}
Nov 03 20:22:09.997510 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:09.997154    4421 scope.go:122] "RemoveContainer" containerID="90e158f7360be80806deefad728af0023158e6310164b18ffd144e8c090481c7"
Nov 03 20:22:10.023088 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.023060    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"]
Nov 03 20:22:10.028704 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.028678    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-1b7b5e-openshift-default-6d7fc588c5-ktnmh"]
Nov 03 20:22:10.215471 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.215437    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z"]
Nov 03 20:22:10.217379 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.217352    4421 cpu_manager.go:414] "containerMap: removing container" podUID="6ef60fe1-39dc-4f56-9114-d7d4e79a6b41" containerName="istio-proxy"
Nov 03 20:22:10.217559 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.217545    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef60fe1-39dc-4f56-9114-d7d4e79a6b41" containerName="istio-proxy"
Nov 03 20:22:10.217699 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.217687    4421 cpu_manager.go:414] "containerMap: removing container" podUID="9ffda400-707f-48d5-9e1f-7f76caf6cd58" containerName="istio-proxy"
Nov 03 20:22:10.217801 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.217789    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ffda400-707f-48d5-9e1f-7f76caf6cd58" containerName="istio-proxy"
Nov 03 20:22:10.217902 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.217892    4421 cpu_manager.go:414] "containerMap: removing container" podUID="6ef60fe1-39dc-4f56-9114-d7d4e79a6b41" containerName="istio-proxy"
Nov 03 20:22:10.217973 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.217962    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef60fe1-39dc-4f56-9114-d7d4e79a6b41" containerName="istio-proxy"
Nov 03 20:22:10.218148 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.218115    4421 cpu_manager.go:414] "containerMap: removing container" podUID="9ffda400-707f-48d5-9e1f-7f76caf6cd58" containerName="istio-proxy"
Nov 03 20:22:10.220050 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.220032    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ffda400-707f-48d5-9e1f-7f76caf6cd58" containerName="istio-proxy"
Nov 03 20:22:10.221152 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.221117    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ffda400-707f-48d5-9e1f-7f76caf6cd58" containerName="istio-proxy"
Nov 03 20:22:10.221440 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.221425    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef60fe1-39dc-4f56-9114-d7d4e79a6b41" containerName="istio-proxy"
Nov 03 20:22:10.221562 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.221552    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef60fe1-39dc-4f56-9114-d7d4e79a6b41" containerName="istio-proxy"
Nov 03 20:22:10.236236 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.234189    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z"]
Nov 03 20:22:10.236236 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.234325    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z"
Nov 03 20:22:10.244177 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.241147    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-efba68-serving-cert\""
Nov 03 20:22:10.260699 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.260665    4421 patch_prober.go:29] interesting pod/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:22:10.260850 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.260735    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn" podUID="249f4a98-95d2-4a3c-9cfc-a6f00e214815" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:22:10.272901 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.272871    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpv4b\" (UniqueName: \"kubernetes.io/projected/e5733b90-6f8c-4105-972f-465f13b91c8a-kube-api-access-qpv4b\") pod \"maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z\" (UID: \"e5733b90-6f8c-4105-972f-465f13b91c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z"
Nov 03 20:22:10.273044 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.272926    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5733b90-6f8c-4105-972f-465f13b91c8a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z\" (UID: \"e5733b90-6f8c-4105-972f-465f13b91c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z"
Nov 03 20:22:10.273044 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.273005    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5733b90-6f8c-4105-972f-465f13b91c8a-metrics-certs\") pod \"maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z\" (UID: \"e5733b90-6f8c-4105-972f-465f13b91c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z"
Nov 03 20:22:10.273196 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.273177    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5733b90-6f8c-4105-972f-465f13b91c8a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z\" (UID: \"e5733b90-6f8c-4105-972f-465f13b91c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z"
Nov 03 20:22:10.374401 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.374373    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpv4b\" (UniqueName: \"kubernetes.io/projected/e5733b90-6f8c-4105-972f-465f13b91c8a-kube-api-access-qpv4b\") pod \"maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z\" (UID: \"e5733b90-6f8c-4105-972f-465f13b91c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z"
Nov 03 20:22:10.374556 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.374419    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5733b90-6f8c-4105-972f-465f13b91c8a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z\" (UID: \"e5733b90-6f8c-4105-972f-465f13b91c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z"
Nov 03 20:22:10.374556 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.374445    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5733b90-6f8c-4105-972f-465f13b91c8a-metrics-certs\") pod \"maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z\" (UID: \"e5733b90-6f8c-4105-972f-465f13b91c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z"
Nov 03 20:22:10.374704 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.374624    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5733b90-6f8c-4105-972f-465f13b91c8a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z\" (UID: \"e5733b90-6f8c-4105-972f-465f13b91c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z"
Nov 03 20:22:10.375231 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.375207    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5733b90-6f8c-4105-972f-465f13b91c8a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z\" (UID: \"e5733b90-6f8c-4105-972f-465f13b91c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z"
Nov 03 20:22:10.377276 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.377254    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5733b90-6f8c-4105-972f-465f13b91c8a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z\" (UID: \"e5733b90-6f8c-4105-972f-465f13b91c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z"
Nov 03 20:22:10.377934 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.377913    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5733b90-6f8c-4105-972f-465f13b91c8a-metrics-certs\") pod \"maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z\" (UID: \"e5733b90-6f8c-4105-972f-465f13b91c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z"
Nov 03 20:22:10.387250 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.387227    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpv4b\" (UniqueName: \"kubernetes.io/projected/e5733b90-6f8c-4105-972f-465f13b91c8a-kube-api-access-qpv4b\") pod \"maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z\" (UID: \"e5733b90-6f8c-4105-972f-465f13b91c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z"
Nov 03 20:22:10.583804 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.558752    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z"
Nov 03 20:22:10.939058 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:10.936186    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z"]
Nov 03 20:22:10.964664 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:22:10.963725    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5733b90_6f8c_4105_972f_465f13b91c8a.slice/crio-4ba42e1af7124a7f3a90579e0781c0509cb6632b6f69d3cf05896a5f9dabe73e WatchSource:0}: Error finding container 4ba42e1af7124a7f3a90579e0781c0509cb6632b6f69d3cf05896a5f9dabe73e: Status 404 returned error can't find the container with id 4ba42e1af7124a7f3a90579e0781c0509cb6632b6f69d3cf05896a5f9dabe73e
Nov 03 20:22:11.005317 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:11.005287    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z" event={"ID":"e5733b90-6f8c-4105-972f-465f13b91c8a","Type":"ContainerStarted","Data":"4ba42e1af7124a7f3a90579e0781c0509cb6632b6f69d3cf05896a5f9dabe73e"}
Nov 03 20:22:11.920744 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:11.920671    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f"
Nov 03 20:22:11.920999 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:11.920982    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5"
Nov 03 20:22:12.028412 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:12.028314    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ffda400-707f-48d5-9e1f-7f76caf6cd58" path="/var/lib/kubelet/pods/9ffda400-707f-48d5-9e1f-7f76caf6cd58/volumes"
Nov 03 20:22:12.028941 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:12.028791    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z" event={"ID":"e5733b90-6f8c-4105-972f-465f13b91c8a","Type":"ContainerStarted","Data":"8aaf06856bbfca477f1ecfc80af762859535787ee3cb73b035150b78ca34cf48"}
Nov 03 20:22:12.028941 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:12.028847    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z"
Nov 03 20:22:12.045767 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:12.045710    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z" podStartSLOduration=1.516509262 podStartE2EDuration="2.045693535s" totalImagesPullingTime="529.184273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:12.044664348 +0000 UTC m=+1262.177792828" watchObservedRunningTime="2026-11-03 20:22:12.045693535 +0000 UTC m=+1262.178821999"
Nov 03 20:22:13.627629 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:13.597689    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2"]
Nov 03 20:22:13.627629 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:13.599067    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ffda400-707f-48d5-9e1f-7f76caf6cd58" containerName="istio-proxy"
Nov 03 20:22:13.822632 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:13.808833    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2"]
Nov 03 20:22:13.822632 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:13.808980    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2"
Nov 03 20:22:13.824666 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:13.823197    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2eee8c\""
Nov 03 20:22:13.831475 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:13.831450    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2eee8c-dockercfg-vjjtm\""
Nov 03 20:22:13.916687 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:13.916582    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7021016a-c1fd-4b0c-849b-7e8470447465-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2\" (UID: \"7021016a-c1fd-4b0c-849b-7e8470447465\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2"
Nov 03 20:22:13.916854 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:13.916708    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q6hk\" (UniqueName: \"kubernetes.io/projected/7021016a-c1fd-4b0c-849b-7e8470447465-kube-api-access-2q6hk\") pod \"payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2\" (UID: \"7021016a-c1fd-4b0c-849b-7e8470447465\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2"
Nov 03 20:22:14.020822 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.020789    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7021016a-c1fd-4b0c-849b-7e8470447465-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2\" (UID: \"7021016a-c1fd-4b0c-849b-7e8470447465\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2"
Nov 03 20:22:14.021407 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.020921    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q6hk\" (UniqueName: \"kubernetes.io/projected/7021016a-c1fd-4b0c-849b-7e8470447465-kube-api-access-2q6hk\") pod \"payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2\" (UID: \"7021016a-c1fd-4b0c-849b-7e8470447465\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2"
Nov 03 20:22:14.022846 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.022816    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7021016a-c1fd-4b0c-849b-7e8470447465-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2\" (UID: \"7021016a-c1fd-4b0c-849b-7e8470447465\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2"
Nov 03 20:22:14.034570 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.034549    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q6hk\" (UniqueName: \"kubernetes.io/projected/7021016a-c1fd-4b0c-849b-7e8470447465-kube-api-access-2q6hk\") pod \"payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2\" (UID: \"7021016a-c1fd-4b0c-849b-7e8470447465\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2"
Nov 03 20:22:14.101449 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.101418    4421 generic.go:356] "Generic (PLEG): container finished" podID="249f4a98-95d2-4a3c-9cfc-a6f00e214815" containerID="c81421f21771c03879aabdfdc40c47e1bd18fd88744c2c3d822738cd0ace0e82" exitCode=0
Nov 03 20:22:14.101588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.101490    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn" event={"ID":"249f4a98-95d2-4a3c-9cfc-a6f00e214815","Type":"ContainerDied","Data":"c81421f21771c03879aabdfdc40c47e1bd18fd88744c2c3d822738cd0ace0e82"}
Nov 03 20:22:14.101588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.101558    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn" event={"ID":"249f4a98-95d2-4a3c-9cfc-a6f00e214815","Type":"ContainerDied","Data":"6e188a11bbe4ddb328fe30c9b33546014bd69eea5529f3293d7ce1c5ebf2d0f9"}
Nov 03 20:22:14.101588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.101584    4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e188a11bbe4ddb328fe30c9b33546014bd69eea5529f3293d7ce1c5ebf2d0f9"
Nov 03 20:22:14.101777 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.101635    4421 scope.go:122] "RemoveContainer" containerID="54c278f026ce64e088b0760f089faf490b21f1ae15859d5a59884713aea1e772"
Nov 03 20:22:14.102767 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.102748    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:22:14.199580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.189794    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2"
Nov 03 20:22:14.266623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.264174    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-token\" (UniqueName: \"kubernetes.io/projected/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-token\") pod \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") "
Nov 03 20:22:14.266623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.264322    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-data\") pod \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") "
Nov 03 20:22:14.266623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.264367    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-envoy\") pod \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") "
Nov 03 20:22:14.266623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.264451    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-credential-socket\") pod \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") "
Nov 03 20:22:14.266623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.264526    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/249f4a98-95d2-4a3c-9cfc-a6f00e214815-kube-api-access-wrmw7\" (UniqueName: \"kubernetes.io/projected/249f4a98-95d2-4a3c-9cfc-a6f00e214815-kube-api-access-wrmw7\") pod \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") "
Nov 03 20:22:14.266623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.264575    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-podinfo\") pod \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") "
Nov 03 20:22:14.266623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.264704    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-workload-socket\") pod \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") "
Nov 03 20:22:14.266623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.264770    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-workload-certs\") pod \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") "
Nov 03 20:22:14.266623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.264889    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istiod-ca-cert\") pod \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\" (UID: \"249f4a98-95d2-4a3c-9cfc-a6f00e214815\") "
Nov 03 20:22:14.266623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.266320    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istiod-ca-cert" pod "249f4a98-95d2-4a3c-9cfc-a6f00e214815" (UID: "249f4a98-95d2-4a3c-9cfc-a6f00e214815"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:22:14.283740 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.283694    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-workload-socket" pod "249f4a98-95d2-4a3c-9cfc-a6f00e214815" (UID: "249f4a98-95d2-4a3c-9cfc-a6f00e214815"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:14.286270 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.286235    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-workload-certs" pod "249f4a98-95d2-4a3c-9cfc-a6f00e214815" (UID: "249f4a98-95d2-4a3c-9cfc-a6f00e214815"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:14.292430 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.292390    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/249f4a98-95d2-4a3c-9cfc-a6f00e214815-kube-api-access-wrmw7" pod "249f4a98-95d2-4a3c-9cfc-a6f00e214815" (UID: "249f4a98-95d2-4a3c-9cfc-a6f00e214815"). InnerVolumeSpecName "kube-api-access-wrmw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:14.293707 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.293682    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-credential-socket" pod "249f4a98-95d2-4a3c-9cfc-a6f00e214815" (UID: "249f4a98-95d2-4a3c-9cfc-a6f00e214815"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:14.295759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.295729    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-envoy" pod "249f4a98-95d2-4a3c-9cfc-a6f00e214815" (UID: "249f4a98-95d2-4a3c-9cfc-a6f00e214815"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:14.295759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.295741    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-podinfo" pod "249f4a98-95d2-4a3c-9cfc-a6f00e214815" (UID: "249f4a98-95d2-4a3c-9cfc-a6f00e214815"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:22:14.295877 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.295757    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-token" pod "249f4a98-95d2-4a3c-9cfc-a6f00e214815" (UID: "249f4a98-95d2-4a3c-9cfc-a6f00e214815"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:14.297843 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.296942    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-data" pod "249f4a98-95d2-4a3c-9cfc-a6f00e214815" (UID: "249f4a98-95d2-4a3c-9cfc-a6f00e214815"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:14.369625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.368925    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:14.369625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.368964    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:14.369625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.368979    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:14.369625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.368996    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:14.369625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.369011    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:14.369625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.369025    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrmw7\" (UniqueName: \"kubernetes.io/projected/249f4a98-95d2-4a3c-9cfc-a6f00e214815-kube-api-access-wrmw7\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:14.369625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.369041    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/249f4a98-95d2-4a3c-9cfc-a6f00e214815-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:14.369625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.369054    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:14.369625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.369068    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/249f4a98-95d2-4a3c-9cfc-a6f00e214815-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:14.665624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:14.641453    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2"]
Nov 03 20:22:14.665624 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:22:14.663694    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7021016a_c1fd_4b0c_849b_7e8470447465.slice/crio-79665edeb2717b3492bfc2bd0874022fdfc8bc2df511bffb58863c4e7c7fb667 WatchSource:0}: Error finding container 79665edeb2717b3492bfc2bd0874022fdfc8bc2df511bffb58863c4e7c7fb667: Status 404 returned error can't find the container with id 79665edeb2717b3492bfc2bd0874022fdfc8bc2df511bffb58863c4e7c7fb667
Nov 03 20:22:15.110778 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:15.110751    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"
Nov 03 20:22:15.112786 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:15.112752    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2" event={"ID":"7021016a-c1fd-4b0c-849b-7e8470447465","Type":"ContainerStarted","Data":"b92644618bd397b3b46d8fc5680aa5dda0b84cc015a58eadadd33bada7a675e9"}
Nov 03 20:22:15.113184 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:15.113160    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2" event={"ID":"7021016a-c1fd-4b0c-849b-7e8470447465","Type":"ContainerStarted","Data":"79665edeb2717b3492bfc2bd0874022fdfc8bc2df511bffb58863c4e7c7fb667"}
Nov 03 20:22:15.172541 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:15.172481    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2" podStartSLOduration=2.172462105 podStartE2EDuration="2.172462105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:15.156408194 +0000 UTC m=+1265.289536659" watchObservedRunningTime="2026-11-03 20:22:15.172462105 +0000 UTC m=+1265.305590568"
Nov 03 20:22:15.183942 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:15.183916    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"]
Nov 03 20:22:15.188932 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:15.188874    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6b9849-openshift-default-6bf96f54dc-2cvtn"]
Nov 03 20:22:15.879902 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:15.879870    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc"]
Nov 03 20:22:15.880412 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:15.880086    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc" podUID="63c38927-79b7-4557-8909-9e792fef6a8c" containerName="maas-api" containerID="cri-o://9ed973fca5c324503834a93bdb3e99aeeacb75ce942c61cceccb414f47a4edc2" gracePeriod=30
Nov 03 20:22:15.884531 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:15.884496    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc" podUID="63c38927-79b7-4557-8909-9e792fef6a8c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.122:8443/health\": dial tcp 10.134.0.122:8443: connect: connection refused"
Nov 03 20:22:16.019718 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:16.019682    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="249f4a98-95d2-4a3c-9cfc-a6f00e214815" path="/var/lib/kubelet/pods/249f4a98-95d2-4a3c-9cfc-a6f00e214815/volumes"
Nov 03 20:22:16.192159 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:16.192127    4421 generic.go:356] "Generic (PLEG): container finished" podID="63c38927-79b7-4557-8909-9e792fef6a8c" containerID="9ed973fca5c324503834a93bdb3e99aeeacb75ce942c61cceccb414f47a4edc2" exitCode=0
Nov 03 20:22:16.192998 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:16.192934    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc" event={"ID":"63c38927-79b7-4557-8909-9e792fef6a8c","Type":"ContainerDied","Data":"9ed973fca5c324503834a93bdb3e99aeeacb75ce942c61cceccb414f47a4edc2"}
Nov 03 20:22:16.193213 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:16.193173    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2"
Nov 03 20:22:16.479274 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:16.479206    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc"
Nov 03 20:22:16.598790 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:16.598759    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63c38927-79b7-4557-8909-9e792fef6a8c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c38927-79b7-4557-8909-9e792fef6a8c-openshift-service-ca\") pod \"63c38927-79b7-4557-8909-9e792fef6a8c\" (UID: \"63c38927-79b7-4557-8909-9e792fef6a8c\") "
Nov 03 20:22:16.598975 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:16.598899    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63c38927-79b7-4557-8909-9e792fef6a8c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c38927-79b7-4557-8909-9e792fef6a8c-metrics-certs\") pod \"63c38927-79b7-4557-8909-9e792fef6a8c\" (UID: \"63c38927-79b7-4557-8909-9e792fef6a8c\") "
Nov 03 20:22:16.598975 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:16.598936    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63c38927-79b7-4557-8909-9e792fef6a8c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c38927-79b7-4557-8909-9e792fef6a8c-maas-api-tls\") pod \"63c38927-79b7-4557-8909-9e792fef6a8c\" (UID: \"63c38927-79b7-4557-8909-9e792fef6a8c\") "
Nov 03 20:22:16.598975 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:16.598956    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c38927-79b7-4557-8909-9e792fef6a8c-kube-api-access-98ndx\" (UniqueName: \"kubernetes.io/projected/63c38927-79b7-4557-8909-9e792fef6a8c-kube-api-access-98ndx\") pod \"63c38927-79b7-4557-8909-9e792fef6a8c\" (UID: \"63c38927-79b7-4557-8909-9e792fef6a8c\") "
Nov 03 20:22:16.599175 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:16.599130    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c38927-79b7-4557-8909-9e792fef6a8c-openshift-service-ca" pod "63c38927-79b7-4557-8909-9e792fef6a8c" (UID: "63c38927-79b7-4557-8909-9e792fef6a8c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:22:16.599611 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:16.599547    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c38927-79b7-4557-8909-9e792fef6a8c-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:16.601284 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:16.601259    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c38927-79b7-4557-8909-9e792fef6a8c-kube-api-access-98ndx" pod "63c38927-79b7-4557-8909-9e792fef6a8c" (UID: "63c38927-79b7-4557-8909-9e792fef6a8c"). InnerVolumeSpecName "kube-api-access-98ndx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:16.601394 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:16.601281    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c38927-79b7-4557-8909-9e792fef6a8c-metrics-certs" pod "63c38927-79b7-4557-8909-9e792fef6a8c" (UID: "63c38927-79b7-4557-8909-9e792fef6a8c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:22:16.601394 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:16.601308    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c38927-79b7-4557-8909-9e792fef6a8c-maas-api-tls" pod "63c38927-79b7-4557-8909-9e792fef6a8c" (UID: "63c38927-79b7-4557-8909-9e792fef6a8c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:22:16.706621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:16.706379    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c38927-79b7-4557-8909-9e792fef6a8c-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:16.706621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:16.706417    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c38927-79b7-4557-8909-9e792fef6a8c-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:16.706621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:16.706436    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98ndx\" (UniqueName: \"kubernetes.io/projected/63c38927-79b7-4557-8909-9e792fef6a8c-kube-api-access-98ndx\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:17.201534 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:17.201489    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc" event={"ID":"63c38927-79b7-4557-8909-9e792fef6a8c","Type":"ContainerDied","Data":"1a49ec05fb8648ebdf6ffda66a6a55690240b31d1cecc78f83f5ce89d5e0c18e"}
Nov 03 20:22:17.202060 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:17.201635    4421 scope.go:122] "RemoveContainer" containerID="9ed973fca5c324503834a93bdb3e99aeeacb75ce942c61cceccb414f47a4edc2"
Nov 03 20:22:17.202060 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:17.201773    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc"
Nov 03 20:22:17.233743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:17.233716    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc"]
Nov 03 20:22:17.237514 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:17.237495    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-400777-7d86648cd5-wh9wc"]
Nov 03 20:22:18.017224 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:18.017193    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63c38927-79b7-4557-8909-9e792fef6a8c" path="/var/lib/kubelet/pods/63c38927-79b7-4557-8909-9e792fef6a8c/volumes"
Nov 03 20:22:18.107661 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:18.107634    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z"
Nov 03 20:22:19.651713 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:22:19.651584    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:22:22.208170 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.208135    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"]
Nov 03 20:22:22.209187 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.209163    4421 cpu_manager.go:414] "containerMap: removing container" podUID="249f4a98-95d2-4a3c-9cfc-a6f00e214815" containerName="istio-proxy"
Nov 03 20:22:22.209272 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.209190    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="249f4a98-95d2-4a3c-9cfc-a6f00e214815" containerName="istio-proxy"
Nov 03 20:22:22.209272 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.209229    4421 cpu_manager.go:414] "containerMap: removing container" podUID="63c38927-79b7-4557-8909-9e792fef6a8c" containerName="maas-api"
Nov 03 20:22:22.209272 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.209237    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c38927-79b7-4557-8909-9e792fef6a8c" containerName="maas-api"
Nov 03 20:22:22.209467 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.209435    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="249f4a98-95d2-4a3c-9cfc-a6f00e214815" containerName="istio-proxy"
Nov 03 20:22:22.209636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.209461    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c38927-79b7-4557-8909-9e792fef6a8c" containerName="maas-api"
Nov 03 20:22:22.214018 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.213443    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.221677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.221196    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-82c116-openshift-default-dockercfg-dgfpt\""
Nov 03 20:22:22.222742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.222717    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"]
Nov 03 20:22:22.253667 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.253627    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c263b0e4-5dd7-4a46-a527-f461b779e42f-istiod-ca-cert\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.253904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.253722    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-workload-certs\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.253904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.253761    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-credential-socket\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.253904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.253883    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbc4f\" (UniqueName: \"kubernetes.io/projected/c263b0e4-5dd7-4a46-a527-f461b779e42f-kube-api-access-pbc4f\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.254143 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.253917    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-data\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.254143 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.253958    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-workload-socket\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.254143 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.254017    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-podinfo\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.254143 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.254047    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-envoy\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.254143 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.254136    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-token\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.357684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.356649    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-workload-certs\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.357684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.356699    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-credential-socket\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.357684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.356766    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbc4f\" (UniqueName: \"kubernetes.io/projected/c263b0e4-5dd7-4a46-a527-f461b779e42f-kube-api-access-pbc4f\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.357684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.356796    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-data\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.357684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.356831    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-workload-socket\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.357684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.356881    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-podinfo\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.357684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.356911    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-envoy\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.357684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.356989    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-token\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.357684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.357144    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c263b0e4-5dd7-4a46-a527-f461b779e42f-istiod-ca-cert\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.390887 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.358029    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c263b0e4-5dd7-4a46-a527-f461b779e42f-istiod-ca-cert\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.390887 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.358317    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-workload-certs\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.390887 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.358542    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-credential-socket\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.390887 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.379526    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-data\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.390887 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.380069    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-workload-socket\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.510670 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.509156    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-token\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.658636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.657261    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-podinfo\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.686619 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.684964    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-envoy\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.698423 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.698019    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbc4f\" (UniqueName: \"kubernetes.io/projected/c263b0e4-5dd7-4a46-a527-f461b779e42f-kube-api-access-pbc4f\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:22.859306 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:22.859212    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:23.208679 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:23.202433    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"]
Nov 03 20:22:23.222229 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:22:23.221466    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc263b0e4_5dd7_4a46_a527_f461b779e42f.slice/crio-28efa2e5042101fd3a800c66a3b7829da664fca2e36bff0b23027aed75120707 WatchSource:0}: Error finding container 28efa2e5042101fd3a800c66a3b7829da664fca2e36bff0b23027aed75120707: Status 404 returned error can't find the container with id 28efa2e5042101fd3a800c66a3b7829da664fca2e36bff0b23027aed75120707
Nov 03 20:22:23.224898 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:23.224861    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:22:23.224998 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:23.224931    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:22:23.224998 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:23.224962    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:22:23.274659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:23.274627    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn" event={"ID":"c263b0e4-5dd7-4a46-a527-f461b779e42f","Type":"ContainerStarted","Data":"28efa2e5042101fd3a800c66a3b7829da664fca2e36bff0b23027aed75120707"}
Nov 03 20:22:24.285003 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:24.284972    4421 generic.go:356] "Generic (PLEG): container finished" podID="f9ec3c59-274d-4af5-a710-751cf5367d3c" containerID="b4cd17d481610745a638bc486c86fee13831ff6e632d37b3f23195c9e777f271" exitCode=0
Nov 03 20:22:24.285490 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:24.285042    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq" event={"ID":"f9ec3c59-274d-4af5-a710-751cf5367d3c","Type":"ContainerDied","Data":"b4cd17d481610745a638bc486c86fee13831ff6e632d37b3f23195c9e777f271"}
Nov 03 20:22:24.285651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:24.285581    4421 scope.go:122] "RemoveContainer" containerID="b4cd17d481610745a638bc486c86fee13831ff6e632d37b3f23195c9e777f271"
Nov 03 20:22:24.287080 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:24.287053    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn" event={"ID":"c263b0e4-5dd7-4a46-a527-f461b779e42f","Type":"ContainerStarted","Data":"01a6879065dc2564a3816015282a6cd62df4f2f7a9817ab678edc8184ca4a60e"}
Nov 03 20:22:24.287371 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:24.287345    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:22:24.287469 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:24.287452    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:22:24.287540 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:24.287509    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:22:24.340430 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:24.340377    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn" podStartSLOduration=2.3403633 podStartE2EDuration="2.3403633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:24.336929864 +0000 UTC m=+1274.470058328" watchObservedRunningTime="2026-11-03 20:22:24.3403633 +0000 UTC m=+1274.473491817"
Nov 03 20:22:24.859892 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:24.859866    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:24.866062 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:24.866037    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:25.304722 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:25.304683    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq" event={"ID":"f9ec3c59-274d-4af5-a710-751cf5367d3c","Type":"ContainerStarted","Data":"25dabfd5874956d61f410ca631a998a8b81437b121d42e494c4e9b3d46e70abc"}
Nov 03 20:22:25.305210 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:25.304792    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:25.306344 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:25.306316    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:22:25.677284 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:25.677182    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:22:25.677474 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:25.677387    4421 patch_prober.go:29] interesting pod/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 20:22:25.677474 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:25.677433    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq" podUID="f9ec3c59-274d-4af5-a710-751cf5367d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 20:22:26.677316 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:26.677264    4421 patch_prober.go:29] interesting pod/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 20:22:26.678242 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:26.677345    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq" podUID="f9ec3c59-274d-4af5-a710-751cf5367d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 20:22:27.213676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:27.211195    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2"
Nov 03 20:22:27.677083 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:27.676996    4421 patch_prober.go:29] interesting pod/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 20:22:27.677083 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:27.677046    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq" podUID="f9ec3c59-274d-4af5-a710-751cf5367d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 20:22:27.964372 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:27.964335    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5"]
Nov 03 20:22:28.221697 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:28.217038    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f"]
Nov 03 20:22:28.682264 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:28.682174    4421 patch_prober.go:29] interesting pod/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 20:22:28.682506 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:28.682473    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq" podUID="f9ec3c59-274d-4af5-a710-751cf5367d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 20:22:29.676930 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:29.676897    4421 patch_prober.go:29] interesting pod/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 20:22:29.677408 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:29.676967    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq" podUID="f9ec3c59-274d-4af5-a710-751cf5367d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 20:22:30.223431 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:30.223393    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq"]
Nov 03 20:22:30.223698 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:30.223669    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq" podUID="25d78e0c-9cfd-46e0-a6cd-a060f352b860" containerName="maas-api" containerID="cri-o://aab65d8778267504af58a483aa46d9979ca1b41764bd2837fdb51e8bf76c119a" gracePeriod=30
Nov 03 20:22:30.238099 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:30.237824    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq" podUID="25d78e0c-9cfd-46e0-a6cd-a060f352b860" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.120:8443/health\": dial tcp 10.134.0.120:8443: connect: connection refused"
Nov 03 20:22:30.424435 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:30.424398    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7"]
Nov 03 20:22:30.424676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:30.424650    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7" podUID="7f3ff7e0-2c72-45c4-bc9c-e58967921176" containerName="payload-processing" containerID="cri-o://9ab9d6fbc35d54ea08a27af5075676edbf29e4f2e452fcbcba97cf3656d78548" gracePeriod=30
Nov 03 20:22:30.490451 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:30.490377    4421 generic.go:356] "Generic (PLEG): container finished" podID="25d78e0c-9cfd-46e0-a6cd-a060f352b860" containerID="aab65d8778267504af58a483aa46d9979ca1b41764bd2837fdb51e8bf76c119a" exitCode=0
Nov 03 20:22:30.490859 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:30.490551    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq" event={"ID":"25d78e0c-9cfd-46e0-a6cd-a060f352b860","Type":"ContainerDied","Data":"aab65d8778267504af58a483aa46d9979ca1b41764bd2837fdb51e8bf76c119a"}
Nov 03 20:22:30.694628 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:30.679039    4421 patch_prober.go:29] interesting pod/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 20:22:30.695649 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:30.695228    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq" podUID="f9ec3c59-274d-4af5-a710-751cf5367d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 20:22:30.999071 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:30.997660    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq"
Nov 03 20:22:31.108622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.108515    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d78e0c-9cfd-46e0-a6cd-a060f352b860-openshift-service-ca" pod "25d78e0c-9cfd-46e0-a6cd-a060f352b860" (UID: "25d78e0c-9cfd-46e0-a6cd-a060f352b860"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:22:31.108791 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.108584    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25d78e0c-9cfd-46e0-a6cd-a060f352b860-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25d78e0c-9cfd-46e0-a6cd-a060f352b860-openshift-service-ca\") pod \"25d78e0c-9cfd-46e0-a6cd-a060f352b860\" (UID: \"25d78e0c-9cfd-46e0-a6cd-a060f352b860\") "
Nov 03 20:22:31.108859 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.108824    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25d78e0c-9cfd-46e0-a6cd-a060f352b860-metrics-certs\" (UniqueName: \"kubernetes.io/secret/25d78e0c-9cfd-46e0-a6cd-a060f352b860-metrics-certs\") pod \"25d78e0c-9cfd-46e0-a6cd-a060f352b860\" (UID: \"25d78e0c-9cfd-46e0-a6cd-a060f352b860\") "
Nov 03 20:22:31.108967 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.108947    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25d78e0c-9cfd-46e0-a6cd-a060f352b860-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25d78e0c-9cfd-46e0-a6cd-a060f352b860-maas-api-tls\") pod \"25d78e0c-9cfd-46e0-a6cd-a060f352b860\" (UID: \"25d78e0c-9cfd-46e0-a6cd-a060f352b860\") "
Nov 03 20:22:31.109051 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.109035    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d78e0c-9cfd-46e0-a6cd-a060f352b860-kube-api-access-2xbq2\" (UniqueName: \"kubernetes.io/projected/25d78e0c-9cfd-46e0-a6cd-a060f352b860-kube-api-access-2xbq2\") pod \"25d78e0c-9cfd-46e0-a6cd-a060f352b860\" (UID: \"25d78e0c-9cfd-46e0-a6cd-a060f352b860\") "
Nov 03 20:22:31.109889 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.109858    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25d78e0c-9cfd-46e0-a6cd-a060f352b860-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:31.111892 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.111853    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d78e0c-9cfd-46e0-a6cd-a060f352b860-kube-api-access-2xbq2" pod "25d78e0c-9cfd-46e0-a6cd-a060f352b860" (UID: "25d78e0c-9cfd-46e0-a6cd-a060f352b860"). InnerVolumeSpecName "kube-api-access-2xbq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:31.112659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.112591    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25d78e0c-9cfd-46e0-a6cd-a060f352b860-metrics-certs" pod "25d78e0c-9cfd-46e0-a6cd-a060f352b860" (UID: "25d78e0c-9cfd-46e0-a6cd-a060f352b860"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:22:31.113514 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.113491    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25d78e0c-9cfd-46e0-a6cd-a060f352b860-maas-api-tls" pod "25d78e0c-9cfd-46e0-a6cd-a060f352b860" (UID: "25d78e0c-9cfd-46e0-a6cd-a060f352b860"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:22:31.185076 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.185050    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7"
Nov 03 20:22:31.210837 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.210807    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f3ff7e0-2c72-45c4-bc9c-e58967921176-kube-api-access-drppz\" (UniqueName: \"kubernetes.io/projected/7f3ff7e0-2c72-45c4-bc9c-e58967921176-kube-api-access-drppz\") pod \"7f3ff7e0-2c72-45c4-bc9c-e58967921176\" (UID: \"7f3ff7e0-2c72-45c4-bc9c-e58967921176\") "
Nov 03 20:22:31.211135 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.211114    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f3ff7e0-2c72-45c4-bc9c-e58967921176-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3ff7e0-2c72-45c4-bc9c-e58967921176-plugins-config-volume\") pod \"7f3ff7e0-2c72-45c4-bc9c-e58967921176\" (UID: \"7f3ff7e0-2c72-45c4-bc9c-e58967921176\") "
Nov 03 20:22:31.211561 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.211531    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f3ff7e0-2c72-45c4-bc9c-e58967921176-plugins-config-volume" pod "7f3ff7e0-2c72-45c4-bc9c-e58967921176" (UID: "7f3ff7e0-2c72-45c4-bc9c-e58967921176"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:22:31.211806 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.211782    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25d78e0c-9cfd-46e0-a6cd-a060f352b860-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:31.211891 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.211811    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25d78e0c-9cfd-46e0-a6cd-a060f352b860-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:31.211891 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.211830    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xbq2\" (UniqueName: \"kubernetes.io/projected/25d78e0c-9cfd-46e0-a6cd-a060f352b860-kube-api-access-2xbq2\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:31.211891 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.211845    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3ff7e0-2c72-45c4-bc9c-e58967921176-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:31.213166 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.213147    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f3ff7e0-2c72-45c4-bc9c-e58967921176-kube-api-access-drppz" pod "7f3ff7e0-2c72-45c4-bc9c-e58967921176" (UID: "7f3ff7e0-2c72-45c4-bc9c-e58967921176"). InnerVolumeSpecName "kube-api-access-drppz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:31.216257 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.216232    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:22:31.323634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.317786    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drppz\" (UniqueName: \"kubernetes.io/projected/7f3ff7e0-2c72-45c4-bc9c-e58967921176-kube-api-access-drppz\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:31.490046 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.490008    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"]
Nov 03 20:22:31.491049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.491024    4421 cpu_manager.go:414] "containerMap: removing container" podUID="7f3ff7e0-2c72-45c4-bc9c-e58967921176" containerName="payload-processing"
Nov 03 20:22:31.491049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.491047    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f3ff7e0-2c72-45c4-bc9c-e58967921176" containerName="payload-processing"
Nov 03 20:22:31.491249 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.491086    4421 cpu_manager.go:414] "containerMap: removing container" podUID="25d78e0c-9cfd-46e0-a6cd-a060f352b860" containerName="maas-api"
Nov 03 20:22:31.491249 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.491093    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d78e0c-9cfd-46e0-a6cd-a060f352b860" containerName="maas-api"
Nov 03 20:22:31.491249 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.491128    4421 cpu_manager.go:414] "containerMap: removing container" podUID="249f4a98-95d2-4a3c-9cfc-a6f00e214815" containerName="istio-proxy"
Nov 03 20:22:31.491249 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.491136    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="249f4a98-95d2-4a3c-9cfc-a6f00e214815" containerName="istio-proxy"
Nov 03 20:22:31.491445 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.491314    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f3ff7e0-2c72-45c4-bc9c-e58967921176" containerName="payload-processing"
Nov 03 20:22:31.491445 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.491333    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d78e0c-9cfd-46e0-a6cd-a060f352b860" containerName="maas-api"
Nov 03 20:22:31.491445 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.491346    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="249f4a98-95d2-4a3c-9cfc-a6f00e214815" containerName="istio-proxy"
Nov 03 20:22:31.495400 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.495372    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:31.500888 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.499942    4421 generic.go:356] "Generic (PLEG): container finished" podID="7f3ff7e0-2c72-45c4-bc9c-e58967921176" containerID="9ab9d6fbc35d54ea08a27af5075676edbf29e4f2e452fcbcba97cf3656d78548" exitCode=0
Nov 03 20:22:31.502934 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.502900    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7" event={"ID":"7f3ff7e0-2c72-45c4-bc9c-e58967921176","Type":"ContainerDied","Data":"9ab9d6fbc35d54ea08a27af5075676edbf29e4f2e452fcbcba97cf3656d78548"}
Nov 03 20:22:31.503056 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.502965    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7" event={"ID":"7f3ff7e0-2c72-45c4-bc9c-e58967921176","Type":"ContainerDied","Data":"bb7678f5c21d83e3680a695c598bf0c51f066dabc7d98018ba44a440d4f16e55"}
Nov 03 20:22:31.503056 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.503000    4421 scope.go:122] "RemoveContainer" containerID="9ab9d6fbc35d54ea08a27af5075676edbf29e4f2e452fcbcba97cf3656d78548"
Nov 03 20:22:31.503193 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.503177    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7"
Nov 03 20:22:31.507541 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.507521    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-3f36ae-openshift-default-dockercfg-fk8qf\""
Nov 03 20:22:31.515073 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.515048    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"]
Nov 03 20:22:31.522295 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.522270    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq" event={"ID":"25d78e0c-9cfd-46e0-a6cd-a060f352b860","Type":"ContainerDied","Data":"1a323a0c1a263d12f8fd076cabc89a2b1cfee21c17b15a97dcc58105bbf14a7f"}
Nov 03 20:22:31.522587 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.522572    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq"
Nov 03 20:22:31.525659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.525581    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8732447-9533-40b3-8da9-e8d6d946785e-istio-token\") pod \"e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") " pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:31.525774 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.525729    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-workload-socket\") pod \"e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") " pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:31.525910 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.525889    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-istio-data\") pod \"e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") " pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:31.526161 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.526146    4421 scope.go:122] "RemoveContainer" containerID="9ab9d6fbc35d54ea08a27af5075676edbf29e4f2e452fcbcba97cf3656d78548"
Nov 03 20:22:31.526963 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.526868    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8732447-9533-40b3-8da9-e8d6d946785e-istiod-ca-cert\") pod \"e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") " pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:31.527120 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.526916    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6dxl\" (UniqueName: \"kubernetes.io/projected/f8732447-9533-40b3-8da9-e8d6d946785e-kube-api-access-w6dxl\") pod \"e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") " pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:31.527702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.527249    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8732447-9533-40b3-8da9-e8d6d946785e-istio-podinfo\") pod \"e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") " pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:31.527702 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:22:31.527267    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ab9d6fbc35d54ea08a27af5075676edbf29e4f2e452fcbcba97cf3656d78548\": container with ID starting with 9ab9d6fbc35d54ea08a27af5075676edbf29e4f2e452fcbcba97cf3656d78548 not found: ID does not exist" containerID="9ab9d6fbc35d54ea08a27af5075676edbf29e4f2e452fcbcba97cf3656d78548"
Nov 03 20:22:31.527702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.527343    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab9d6fbc35d54ea08a27af5075676edbf29e4f2e452fcbcba97cf3656d78548"} err="failed to get container status \"9ab9d6fbc35d54ea08a27af5075676edbf29e4f2e452fcbcba97cf3656d78548\": rpc error: code = NotFound desc = could not find container \"9ab9d6fbc35d54ea08a27af5075676edbf29e4f2e452fcbcba97cf3656d78548\": container with ID starting with 9ab9d6fbc35d54ea08a27af5075676edbf29e4f2e452fcbcba97cf3656d78548 not found: ID does not exist"
Nov 03 20:22:31.527702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.527372    4421 scope.go:122] "RemoveContainer" containerID="aab65d8778267504af58a483aa46d9979ca1b41764bd2837fdb51e8bf76c119a"
Nov 03 20:22:31.527702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.527567    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-workload-certs\") pod \"e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") " pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:31.527702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.527660    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-credential-socket\") pod \"e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") " pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:31.528126 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.527713    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-istio-envoy\") pod \"e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") " pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:31.581979 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.581886    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7"]
Nov 03 20:22:31.644624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.640745    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-workload-socket\") pod \"e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") " pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:31.644624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.640890    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-istio-data\") pod \"e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") " pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:31.644624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.640969    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8732447-9533-40b3-8da9-e8d6d946785e-istiod-ca-cert\") pod \"e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") " pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:31.644624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.641004    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6dxl\" (UniqueName: \"kubernetes.io/projected/f8732447-9533-40b3-8da9-e8d6d946785e-kube-api-access-w6dxl\") pod \"e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") " pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:31.644624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.641071    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8732447-9533-40b3-8da9-e8d6d946785e-istio-podinfo\") pod \"e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") " pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:31.644624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.641194    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-workload-certs\") pod \"e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") " pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:31.644624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.641230    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-credential-socket\") pod \"e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") " pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:31.644624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.641290    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-istio-envoy\") pod \"e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") " pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:31.644624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.641354    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8732447-9533-40b3-8da9-e8d6d946785e-istio-token\") pod \"e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") " pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:31.644624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.642449    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-workload-socket\") pod \"e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") " pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:31.644624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.642715    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-istio-data\") pod \"e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") " pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:31.644624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.643327    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8732447-9533-40b3-8da9-e8d6d946785e-istiod-ca-cert\") pod \"e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") " pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:31.646133 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.646111    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-workload-certs\") pod \"e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") " pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:31.646880 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.646855    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-credential-socket\") pod \"e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") " pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:31.669797 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.669770    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b07e26-667f9bbb68-rb9v7"]
Nov 03 20:22:31.682041 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.682008    4421 patch_prober.go:29] interesting pod/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 20:22:31.682304 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.682277    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq" podUID="f9ec3c59-274d-4af5-a710-751cf5367d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 20:22:31.757674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.753977    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-istio-envoy\") pod \"e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") " pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:31.757674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.755285    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8732447-9533-40b3-8da9-e8d6d946785e-istio-podinfo\") pod \"e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") " pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:31.790230 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.790184    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq"]
Nov 03 20:22:31.816860 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.816818    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b07e26-67865b9bb5-wzpvq"]
Nov 03 20:22:31.821664 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.821583    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8732447-9533-40b3-8da9-e8d6d946785e-istio-token\") pod \"e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") " pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:31.839937 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:31.839907    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6dxl\" (UniqueName: \"kubernetes.io/projected/f8732447-9533-40b3-8da9-e8d6d946785e-kube-api-access-w6dxl\") pod \"e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") " pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:32.021579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:32.021501    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25d78e0c-9cfd-46e0-a6cd-a060f352b860" path="/var/lib/kubelet/pods/25d78e0c-9cfd-46e0-a6cd-a060f352b860/volumes"
Nov 03 20:22:32.022300 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:32.022262    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f3ff7e0-2c72-45c4-bc9c-e58967921176" path="/var/lib/kubelet/pods/7f3ff7e0-2c72-45c4-bc9c-e58967921176/volumes"
Nov 03 20:22:32.132948 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:32.132916    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:32.439931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:32.435625    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"]
Nov 03 20:22:32.457216 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:32.457139    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:22:32.457661 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:32.457625    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:22:32.457948 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:32.457921    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:22:32.533339 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:32.533310    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9" event={"ID":"f8732447-9533-40b3-8da9-e8d6d946785e","Type":"ContainerStarted","Data":"f4c583c9d80f34875fa28c297d3e0bd2a28e6df0cfb6624c22ee62762aeb9b52"}
Nov 03 20:22:32.683079 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:32.681834    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:22:32.683231 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:32.683215    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:22:32.969671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:32.969634    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5" podUID="11a98991-daa5-43ef-ad1d-e378920a76e5" containerName="payload-processing" containerID="cri-o://6690e7e83863665717f8ee04a496da3b0f105a090e8cf973d7eb2f3e30a5456e" gracePeriod=25
Nov 03 20:22:33.217900 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.217859    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f" podUID="a4345985-0271-49fe-8cfa-8478d88209c7" containerName="payload-pre-processing" containerID="cri-o://1efd2e948b8102edfa8619f77ae069474c955de4c8253c22ee30c8284463c4a1" gracePeriod=25
Nov 03 20:22:33.361796 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.361771    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5"
Nov 03 20:22:33.425714 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.421186    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a98991-daa5-43ef-ad1d-e378920a76e5-kube-api-access-892t9\" (UniqueName: \"kubernetes.io/projected/11a98991-daa5-43ef-ad1d-e378920a76e5-kube-api-access-892t9\") pod \"11a98991-daa5-43ef-ad1d-e378920a76e5\" (UID: \"11a98991-daa5-43ef-ad1d-e378920a76e5\") "
Nov 03 20:22:33.425714 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.421400    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11a98991-daa5-43ef-ad1d-e378920a76e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a98991-daa5-43ef-ad1d-e378920a76e5-plugins-config-volume\") pod \"11a98991-daa5-43ef-ad1d-e378920a76e5\" (UID: \"11a98991-daa5-43ef-ad1d-e378920a76e5\") "
Nov 03 20:22:33.450754 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.449849    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a98991-daa5-43ef-ad1d-e378920a76e5-plugins-config-volume" pod "11a98991-daa5-43ef-ad1d-e378920a76e5" (UID: "11a98991-daa5-43ef-ad1d-e378920a76e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:22:33.495705 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.484527    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a98991-daa5-43ef-ad1d-e378920a76e5-kube-api-access-892t9" pod "11a98991-daa5-43ef-ad1d-e378920a76e5" (UID: "11a98991-daa5-43ef-ad1d-e378920a76e5"). InnerVolumeSpecName "kube-api-access-892t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:33.524023 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.523993    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a98991-daa5-43ef-ad1d-e378920a76e5-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:33.524023 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.524023    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-892t9\" (UniqueName: \"kubernetes.io/projected/11a98991-daa5-43ef-ad1d-e378920a76e5-kube-api-access-892t9\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:33.544851 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.544576    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9" event={"ID":"f8732447-9533-40b3-8da9-e8d6d946785e","Type":"ContainerStarted","Data":"73120e7fc615b4e82e5509496d1f0d1f8614ff615bd7e4efc2a949681dbc8f11"}
Nov 03 20:22:33.546204 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.546175    4421 generic.go:356] "Generic (PLEG): container finished" podID="11a98991-daa5-43ef-ad1d-e378920a76e5" containerID="6690e7e83863665717f8ee04a496da3b0f105a090e8cf973d7eb2f3e30a5456e" exitCode=0
Nov 03 20:22:33.546321 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.546297    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5" event={"ID":"11a98991-daa5-43ef-ad1d-e378920a76e5","Type":"ContainerDied","Data":"6690e7e83863665717f8ee04a496da3b0f105a090e8cf973d7eb2f3e30a5456e"}
Nov 03 20:22:33.546364 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.546336    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5" event={"ID":"11a98991-daa5-43ef-ad1d-e378920a76e5","Type":"ContainerDied","Data":"41624f519acd2aac0c0ab52b2dd5651c8dfa7fb7a48cf2680c02f37ac04cc4f0"}
Nov 03 20:22:33.546400 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.546364    4421 scope.go:122] "RemoveContainer" containerID="6690e7e83863665717f8ee04a496da3b0f105a090e8cf973d7eb2f3e30a5456e"
Nov 03 20:22:33.546498 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.546483    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5"
Nov 03 20:22:33.551804 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.551774    4421 generic.go:356] "Generic (PLEG): container finished" podID="a4345985-0271-49fe-8cfa-8478d88209c7" containerID="1efd2e948b8102edfa8619f77ae069474c955de4c8253c22ee30c8284463c4a1" exitCode=0
Nov 03 20:22:33.552973 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.552839    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f" event={"ID":"a4345985-0271-49fe-8cfa-8478d88209c7","Type":"ContainerDied","Data":"1efd2e948b8102edfa8619f77ae069474c955de4c8253c22ee30c8284463c4a1"}
Nov 03 20:22:33.563758 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.563717    4421 scope.go:122] "RemoveContainer" containerID="6690e7e83863665717f8ee04a496da3b0f105a090e8cf973d7eb2f3e30a5456e"
Nov 03 20:22:33.564120 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:22:33.564099    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6690e7e83863665717f8ee04a496da3b0f105a090e8cf973d7eb2f3e30a5456e\": container with ID starting with 6690e7e83863665717f8ee04a496da3b0f105a090e8cf973d7eb2f3e30a5456e not found: ID does not exist" containerID="6690e7e83863665717f8ee04a496da3b0f105a090e8cf973d7eb2f3e30a5456e"
Nov 03 20:22:33.564211 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.564127    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6690e7e83863665717f8ee04a496da3b0f105a090e8cf973d7eb2f3e30a5456e"} err="failed to get container status \"6690e7e83863665717f8ee04a496da3b0f105a090e8cf973d7eb2f3e30a5456e\": rpc error: code = NotFound desc = could not find container \"6690e7e83863665717f8ee04a496da3b0f105a090e8cf973d7eb2f3e30a5456e\": container with ID starting with 6690e7e83863665717f8ee04a496da3b0f105a090e8cf973d7eb2f3e30a5456e not found: ID does not exist"
Nov 03 20:22:33.575270 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.575225    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9" podStartSLOduration=2.575211878 podStartE2EDuration="2.575211878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:33.572338017 +0000 UTC m=+1283.705466491" watchObservedRunningTime="2026-11-03 20:22:33.575211878 +0000 UTC m=+1283.708340340"
Nov 03 20:22:33.591265 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.591223    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5"]
Nov 03 20:22:33.596958 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.596935    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efba68-69596d49cd-c72m5"]
Nov 03 20:22:33.632205 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.632183    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f"
Nov 03 20:22:33.734881 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.732415    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4345985-0271-49fe-8cfa-8478d88209c7-kube-api-access-wxmwb\" (UniqueName: \"kubernetes.io/projected/a4345985-0271-49fe-8cfa-8478d88209c7-kube-api-access-wxmwb\") pod \"a4345985-0271-49fe-8cfa-8478d88209c7\" (UID: \"a4345985-0271-49fe-8cfa-8478d88209c7\") "
Nov 03 20:22:33.760742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.757723    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4345985-0271-49fe-8cfa-8478d88209c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4345985-0271-49fe-8cfa-8478d88209c7-plugins-config-volume\") pod \"a4345985-0271-49fe-8cfa-8478d88209c7\" (UID: \"a4345985-0271-49fe-8cfa-8478d88209c7\") "
Nov 03 20:22:33.780731 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.775255    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4345985-0271-49fe-8cfa-8478d88209c7-plugins-config-volume" pod "a4345985-0271-49fe-8cfa-8478d88209c7" (UID: "a4345985-0271-49fe-8cfa-8478d88209c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:22:33.780896 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.780817    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4345985-0271-49fe-8cfa-8478d88209c7-kube-api-access-wxmwb" pod "a4345985-0271-49fe-8cfa-8478d88209c7" (UID: "a4345985-0271-49fe-8cfa-8478d88209c7"). InnerVolumeSpecName "kube-api-access-wxmwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:33.860027 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.859995    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxmwb\" (UniqueName: \"kubernetes.io/projected/a4345985-0271-49fe-8cfa-8478d88209c7-kube-api-access-wxmwb\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:33.860027 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:33.860020    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4345985-0271-49fe-8cfa-8478d88209c7-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:34.016881 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:34.016802    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11a98991-daa5-43ef-ad1d-e378920a76e5" path="/var/lib/kubelet/pods/11a98991-daa5-43ef-ad1d-e378920a76e5/volumes"
Nov 03 20:22:34.133263 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:34.133225    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:34.139086 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:34.139059    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:34.642622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:34.642156    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f"
Nov 03 20:22:34.668624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:34.642834    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f" event={"ID":"a4345985-0271-49fe-8cfa-8478d88209c7","Type":"ContainerDied","Data":"7127e5bdc77fa5c06f2767ce541f1de9e39d17f5790e7b1e4609726c2e25d1a2"}
Nov 03 20:22:34.668624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:34.642888    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:34.668624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:34.642920    4421 scope.go:122] "RemoveContainer" containerID="1efd2e948b8102edfa8619f77ae069474c955de4c8253c22ee30c8284463c4a1"
Nov 03 20:22:34.668624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:34.658238    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:22:34.834223 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:34.834100    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f"]
Nov 03 20:22:34.846447 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:34.838852    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efba68-75766bf75-sxn6f"]
Nov 03 20:22:35.861724 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:35.861690    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm"]
Nov 03 20:22:35.862720 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:35.862688    4421 cpu_manager.go:414] "containerMap: removing container" podUID="a4345985-0271-49fe-8cfa-8478d88209c7" containerName="payload-pre-processing"
Nov 03 20:22:35.862720 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:35.862712    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4345985-0271-49fe-8cfa-8478d88209c7" containerName="payload-pre-processing"
Nov 03 20:22:35.862922 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:35.862726    4421 cpu_manager.go:414] "containerMap: removing container" podUID="11a98991-daa5-43ef-ad1d-e378920a76e5" containerName="payload-processing"
Nov 03 20:22:35.862922 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:35.862733    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a98991-daa5-43ef-ad1d-e378920a76e5" containerName="payload-processing"
Nov 03 20:22:35.862922 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:35.862905    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4345985-0271-49fe-8cfa-8478d88209c7" containerName="payload-pre-processing"
Nov 03 20:22:35.862922 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:35.862918    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a98991-daa5-43ef-ad1d-e378920a76e5" containerName="payload-processing"
Nov 03 20:22:35.868184 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:35.868161    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm"
Nov 03 20:22:35.873610 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:35.873568    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1142fd\""
Nov 03 20:22:35.880005 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:35.879975    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm"]
Nov 03 20:22:35.901641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:35.901587    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp"]
Nov 03 20:22:35.906979 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:35.906933    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp"
Nov 03 20:22:35.917216 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:35.913106    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1142fd-dockercfg-bpzjt\""
Nov 03 20:22:35.917216 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:35.914964    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp"]
Nov 03 20:22:36.092620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:36.085361    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp\" (UID: \"24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp"
Nov 03 20:22:36.092620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:36.085429    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvvdk\" (UniqueName: \"kubernetes.io/projected/54bdf951-8d0e-4acc-bb6a-75cb74e2ae98-kube-api-access-zvvdk\") pod \"payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm\" (UID: \"54bdf951-8d0e-4acc-bb6a-75cb74e2ae98\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm"
Nov 03 20:22:36.092620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:36.085583    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsr2v\" (UniqueName: \"kubernetes.io/projected/24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c-kube-api-access-vsr2v\") pod \"payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp\" (UID: \"24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp"
Nov 03 20:22:36.092620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:36.085704    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54bdf951-8d0e-4acc-bb6a-75cb74e2ae98-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm\" (UID: \"54bdf951-8d0e-4acc-bb6a-75cb74e2ae98\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm"
Nov 03 20:22:36.178672 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:36.172563    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4345985-0271-49fe-8cfa-8478d88209c7" path="/var/lib/kubelet/pods/a4345985-0271-49fe-8cfa-8478d88209c7/volumes"
Nov 03 20:22:36.198624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:36.193925    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54bdf951-8d0e-4acc-bb6a-75cb74e2ae98-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm\" (UID: \"54bdf951-8d0e-4acc-bb6a-75cb74e2ae98\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm"
Nov 03 20:22:36.198624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:36.194035    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp\" (UID: \"24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp"
Nov 03 20:22:36.198624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:36.194092    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvvdk\" (UniqueName: \"kubernetes.io/projected/54bdf951-8d0e-4acc-bb6a-75cb74e2ae98-kube-api-access-zvvdk\") pod \"payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm\" (UID: \"54bdf951-8d0e-4acc-bb6a-75cb74e2ae98\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm"
Nov 03 20:22:36.198624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:36.194250    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsr2v\" (UniqueName: \"kubernetes.io/projected/24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c-kube-api-access-vsr2v\") pod \"payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp\" (UID: \"24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp"
Nov 03 20:22:36.198624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:36.195210    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp\" (UID: \"24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp"
Nov 03 20:22:36.198624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:36.196211    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54bdf951-8d0e-4acc-bb6a-75cb74e2ae98-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm\" (UID: \"54bdf951-8d0e-4acc-bb6a-75cb74e2ae98\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm"
Nov 03 20:22:36.248623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:36.242504    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvvdk\" (UniqueName: \"kubernetes.io/projected/54bdf951-8d0e-4acc-bb6a-75cb74e2ae98-kube-api-access-zvvdk\") pod \"payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm\" (UID: \"54bdf951-8d0e-4acc-bb6a-75cb74e2ae98\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm"
Nov 03 20:22:36.263638 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:36.262391    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsr2v\" (UniqueName: \"kubernetes.io/projected/24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c-kube-api-access-vsr2v\") pod \"payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp\" (UID: \"24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp"
Nov 03 20:22:36.496621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:36.489885    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm"
Nov 03 20:22:36.534437 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:36.534279    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp"
Nov 03 20:22:36.844744 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:22:36.844716    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54bdf951_8d0e_4acc_bb6a_75cb74e2ae98.slice/crio-609e01486f6ba74101fa8d71913bd30356b3b96a1be55882838d84cf1faacd53 WatchSource:0}: Error finding container 609e01486f6ba74101fa8d71913bd30356b3b96a1be55882838d84cf1faacd53: Status 404 returned error can't find the container with id 609e01486f6ba74101fa8d71913bd30356b3b96a1be55882838d84cf1faacd53
Nov 03 20:22:36.844981 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:36.844960    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm"]
Nov 03 20:22:36.889932 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:36.889893    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp"]
Nov 03 20:22:36.899957 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:22:36.899934    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ada5c2_1dd7_442b_9ab2_b8e6e3b2f97c.slice/crio-b1f258370484585000a848559320643331c7beab57e7e7716b58b11bec6d0441 WatchSource:0}: Error finding container b1f258370484585000a848559320643331c7beab57e7e7716b58b11bec6d0441: Status 404 returned error can't find the container with id b1f258370484585000a848559320643331c7beab57e7e7716b58b11bec6d0441
Nov 03 20:22:37.374286 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.374254    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj"]
Nov 03 20:22:37.379120 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.379100    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj"
Nov 03 20:22:37.386394 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.386356    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-3f36ae\""
Nov 03 20:22:37.391758 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.391730    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj"]
Nov 03 20:22:37.402778 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.402746    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-3f36ae-dockercfg-tzgjn\""
Nov 03 20:22:37.419449 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.419418    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6"]
Nov 03 20:22:37.425289 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.425258    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6"
Nov 03 20:22:37.426039 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.426011    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x4q8\" (UniqueName: \"kubernetes.io/projected/b3dbca1e-d679-4e9e-960d-9006b92b6d72-kube-api-access-2x4q8\") pod \"payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj\" (UID: \"b3dbca1e-d679-4e9e-960d-9006b92b6d72\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj"
Nov 03 20:22:37.426218 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.426198    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3dbca1e-d679-4e9e-960d-9006b92b6d72-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj\" (UID: \"b3dbca1e-d679-4e9e-960d-9006b92b6d72\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj"
Nov 03 20:22:37.433793 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.433743    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6"]
Nov 03 20:22:37.605655 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.561768    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3dbca1e-d679-4e9e-960d-9006b92b6d72-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj\" (UID: \"b3dbca1e-d679-4e9e-960d-9006b92b6d72\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj"
Nov 03 20:22:37.605655 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.561845    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6\" (UID: \"6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6"
Nov 03 20:22:37.605655 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.561968    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx647\" (UniqueName: \"kubernetes.io/projected/6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8-kube-api-access-hx647\") pod \"payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6\" (UID: \"6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6"
Nov 03 20:22:37.605655 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.562158    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x4q8\" (UniqueName: \"kubernetes.io/projected/b3dbca1e-d679-4e9e-960d-9006b92b6d72-kube-api-access-2x4q8\") pod \"payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj\" (UID: \"b3dbca1e-d679-4e9e-960d-9006b92b6d72\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj"
Nov 03 20:22:37.605655 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.598518    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3dbca1e-d679-4e9e-960d-9006b92b6d72-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj\" (UID: \"b3dbca1e-d679-4e9e-960d-9006b92b6d72\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj"
Nov 03 20:22:37.669670 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.664976    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6\" (UID: \"6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6"
Nov 03 20:22:37.669670 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.665105    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx647\" (UniqueName: \"kubernetes.io/projected/6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8-kube-api-access-hx647\") pod \"payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6\" (UID: \"6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6"
Nov 03 20:22:37.669670 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.666853    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6\" (UID: \"6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6"
Nov 03 20:22:37.699389 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.699309    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx647\" (UniqueName: \"kubernetes.io/projected/6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8-kube-api-access-hx647\") pod \"payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6\" (UID: \"6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6"
Nov 03 20:22:37.704634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.700748    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x4q8\" (UniqueName: \"kubernetes.io/projected/b3dbca1e-d679-4e9e-960d-9006b92b6d72-kube-api-access-2x4q8\") pod \"payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj\" (UID: \"b3dbca1e-d679-4e9e-960d-9006b92b6d72\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj"
Nov 03 20:22:37.747891 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.747856    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6"
Nov 03 20:22:37.861619 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.860795    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm" event={"ID":"54bdf951-8d0e-4acc-bb6a-75cb74e2ae98","Type":"ContainerStarted","Data":"e7dc80838428d7d8687d9117f9222295436f3595888b3f3d3f868a892f87ca6a"}
Nov 03 20:22:37.861619 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.860850    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm" event={"ID":"54bdf951-8d0e-4acc-bb6a-75cb74e2ae98","Type":"ContainerStarted","Data":"609e01486f6ba74101fa8d71913bd30356b3b96a1be55882838d84cf1faacd53"}
Nov 03 20:22:37.861619 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.861109    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm"
Nov 03 20:22:37.863359 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.863338    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp" event={"ID":"24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c","Type":"ContainerStarted","Data":"909b400ced035fcd5228faabfc8d3b7f7507455fc89b03bb4f6c3415e6e09878"}
Nov 03 20:22:37.863490 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.863381    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp" event={"ID":"24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c","Type":"ContainerStarted","Data":"b1f258370484585000a848559320643331c7beab57e7e7716b58b11bec6d0441"}
Nov 03 20:22:37.863612 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.863578    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp"
Nov 03 20:22:37.882780 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.882722    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm" podStartSLOduration=2.882706422 podStartE2EDuration="2.882706422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:37.878241637 +0000 UTC m=+1288.011370096" watchObservedRunningTime="2026-11-03 20:22:37.882706422 +0000 UTC m=+1288.015834885"
Nov 03 20:22:37.905215 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.905155    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp" podStartSLOduration=2.905135755 podStartE2EDuration="2.905135755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:37.89729679 +0000 UTC m=+1288.030425253" watchObservedRunningTime="2026-11-03 20:22:37.905135755 +0000 UTC m=+1288.038264217"
Nov 03 20:22:37.995648 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:37.995609    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj"
Nov 03 20:22:38.118962 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.118933    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6"]
Nov 03 20:22:38.121760 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:22:38.121725    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dfee168_d0d0_4f82_bcc3_3a2bf0b0b1d8.slice/crio-a8465b6101c8806f35c8de76d3000dd2465f80e40f99c4a16cacc569286f4057 WatchSource:0}: Error finding container a8465b6101c8806f35c8de76d3000dd2465f80e40f99c4a16cacc569286f4057: Status 404 returned error can't find the container with id a8465b6101c8806f35c8de76d3000dd2465f80e40f99c4a16cacc569286f4057
Nov 03 20:22:38.205177 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.205146    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn"]
Nov 03 20:22:38.214294 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.214268    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn"
Nov 03 20:22:38.223312 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.223283    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn"]
Nov 03 20:22:38.223516 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.223325    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-82c116-serving-cert\""
Nov 03 20:22:38.290509 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.290480    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e81ae3d3-e34d-4c9b-8f0b-739204ede414-maas-api-tls\") pod \"maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn\" (UID: \"e81ae3d3-e34d-4c9b-8f0b-739204ede414\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn"
Nov 03 20:22:38.290692 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.290539    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w77fr\" (UniqueName: \"kubernetes.io/projected/e81ae3d3-e34d-4c9b-8f0b-739204ede414-kube-api-access-w77fr\") pod \"maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn\" (UID: \"e81ae3d3-e34d-4c9b-8f0b-739204ede414\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn"
Nov 03 20:22:38.290894 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.290867    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e81ae3d3-e34d-4c9b-8f0b-739204ede414-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn\" (UID: \"e81ae3d3-e34d-4c9b-8f0b-739204ede414\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn"
Nov 03 20:22:38.290993 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.290979    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e81ae3d3-e34d-4c9b-8f0b-739204ede414-metrics-certs\") pod \"maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn\" (UID: \"e81ae3d3-e34d-4c9b-8f0b-739204ede414\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn"
Nov 03 20:22:38.406623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.394561    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj"]
Nov 03 20:22:38.406623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.405554    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e81ae3d3-e34d-4c9b-8f0b-739204ede414-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn\" (UID: \"e81ae3d3-e34d-4c9b-8f0b-739204ede414\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn"
Nov 03 20:22:38.406944 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.406735    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e81ae3d3-e34d-4c9b-8f0b-739204ede414-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn\" (UID: \"e81ae3d3-e34d-4c9b-8f0b-739204ede414\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn"
Nov 03 20:22:38.406944 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.406827    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e81ae3d3-e34d-4c9b-8f0b-739204ede414-metrics-certs\") pod \"maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn\" (UID: \"e81ae3d3-e34d-4c9b-8f0b-739204ede414\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn"
Nov 03 20:22:38.410620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.407452    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e81ae3d3-e34d-4c9b-8f0b-739204ede414-maas-api-tls\") pod \"maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn\" (UID: \"e81ae3d3-e34d-4c9b-8f0b-739204ede414\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn"
Nov 03 20:22:38.410620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.409052    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w77fr\" (UniqueName: \"kubernetes.io/projected/e81ae3d3-e34d-4c9b-8f0b-739204ede414-kube-api-access-w77fr\") pod \"maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn\" (UID: \"e81ae3d3-e34d-4c9b-8f0b-739204ede414\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn"
Nov 03 20:22:38.456933 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.456857    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w77fr\" (UniqueName: \"kubernetes.io/projected/e81ae3d3-e34d-4c9b-8f0b-739204ede414-kube-api-access-w77fr\") pod \"maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn\" (UID: \"e81ae3d3-e34d-4c9b-8f0b-739204ede414\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn"
Nov 03 20:22:38.460500 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.460440    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e81ae3d3-e34d-4c9b-8f0b-739204ede414-maas-api-tls\") pod \"maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn\" (UID: \"e81ae3d3-e34d-4c9b-8f0b-739204ede414\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn"
Nov 03 20:22:38.475620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.471101    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e81ae3d3-e34d-4c9b-8f0b-739204ede414-metrics-certs\") pod \"maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn\" (UID: \"e81ae3d3-e34d-4c9b-8f0b-739204ede414\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn"
Nov 03 20:22:38.556433 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.556399    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn"
Nov 03 20:22:38.912540 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.911064    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn"]
Nov 03 20:22:38.926520 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.926493    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj" event={"ID":"b3dbca1e-d679-4e9e-960d-9006b92b6d72","Type":"ContainerStarted","Data":"27e621c3ee88618bf51812cd8b1c6758ee75a93d615adb282921e5193946a459"}
Nov 03 20:22:38.926744 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.926724    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj" event={"ID":"b3dbca1e-d679-4e9e-960d-9006b92b6d72","Type":"ContainerStarted","Data":"c422246fac0446bdc2b92b407befb7b920c4fbbaf462bf90532ef687e7db42a2"}
Nov 03 20:22:38.926874 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.926861    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj"
Nov 03 20:22:38.931010 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.930228    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6" event={"ID":"6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8","Type":"ContainerStarted","Data":"6dc54b7e81966bf70991beb151eca12e307b88340845ae711960b12a3c3fb230"}
Nov 03 20:22:38.931010 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.930276    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6"
Nov 03 20:22:38.931010 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.930294    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6" event={"ID":"6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8","Type":"ContainerStarted","Data":"a8465b6101c8806f35c8de76d3000dd2465f80e40f99c4a16cacc569286f4057"}
Nov 03 20:22:38.954446 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.954387    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj" podStartSLOduration=1.954367787 podStartE2EDuration="1.954367787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:38.950855904 +0000 UTC m=+1289.083984367" watchObservedRunningTime="2026-11-03 20:22:38.954367787 +0000 UTC m=+1289.087496251"
Nov 03 20:22:38.972139 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:38.972062    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6" podStartSLOduration=1.972043243 podStartE2EDuration="1.972043243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:38.969701512 +0000 UTC m=+1289.102829994" watchObservedRunningTime="2026-11-03 20:22:38.972043243 +0000 UTC m=+1289.105171705"
Nov 03 20:22:39.938873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:39.938835    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn" event={"ID":"e81ae3d3-e34d-4c9b-8f0b-739204ede414","Type":"ContainerStarted","Data":"350c013f72ae603424a12c6d2cafaceba8b72fdd6a2202ae95866df237f1ed79"}
Nov 03 20:22:39.939292 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:39.938886    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn" event={"ID":"e81ae3d3-e34d-4c9b-8f0b-739204ede414","Type":"ContainerStarted","Data":"d8bf4400d9d10b81831840322b5ccb8f068630e46dbad680c6ea2959c0e0c8b9"}
Nov 03 20:22:39.962346 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:39.962299    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn" podStartSLOduration=1.3620554409999999 podStartE2EDuration="1.962285192s" totalImagesPullingTime="600.229751ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:39.960003521 +0000 UTC m=+1290.093131983" watchObservedRunningTime="2026-11-03 20:22:39.962285192 +0000 UTC m=+1290.095413654"
Nov 03 20:22:40.026623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:40.025638    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"]
Nov 03 20:22:40.026623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:40.025843    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh" podUID="10a12293-4a77-4829-93da-7e1909f3853f" containerName="istio-proxy" containerID="cri-o://a9c524e56122a534d3b85a6391912e7e74d6bdf7af3c6b0a6fae7f752170c8ed" gracePeriod=30
Nov 03 20:22:40.947111 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:40.947071    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn"
Nov 03 20:22:44.030860 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:44.030828    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"]
Nov 03 20:22:44.031336 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:44.031026    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z" podUID="34c80431-a853-4012-b239-ea4659c70bf9" containerName="istio-proxy" containerID="cri-o://84c1151db91f57fbd6758c31846f3666bc635f87815afe117e56eac0220901db" gracePeriod=30
Nov 03 20:22:44.035026 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:44.035001    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"]
Nov 03 20:22:44.035225 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:44.035203    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm" podUID="08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc" containerName="istio-proxy" containerID="cri-o://bde8000b904e5f5f581abdb127e2a365a8b36201d2e4dfc6815a921bb8d05cfe" gracePeriod=30
Nov 03 20:22:45.102761 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.102731    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"]
Nov 03 20:22:45.103163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.102959    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5" podUID="7bb10f7c-7a24-482b-961e-966a83c8139d" containerName="istio-proxy" containerID="cri-o://cc649c45aff40c543c3d7cb57f11823f915340bf656528282609bb229d9c3af0" gracePeriod=30
Nov 03 20:22:45.107058 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.107034    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"]
Nov 03 20:22:45.107277 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.107247    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb" podUID="8f6f3683-cb02-4dd2-aed6-a919a84c2e4c" containerName="istio-proxy" containerID="cri-o://49d100bfcb4c65564d1f8d4e077dcbac06ab79aa87b82ae2fa63d75be59ed94a" gracePeriod=30
Nov 03 20:22:45.669955 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.667923    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:22:45.703590 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.703559    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-istio-data\") pod \"10a12293-4a77-4829-93da-7e1909f3853f\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") "
Nov 03 20:22:45.703797 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.703643    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-workload-socket\") pod \"10a12293-4a77-4829-93da-7e1909f3853f\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") "
Nov 03 20:22:45.703797 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.703728    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-workload-certs\") pod \"10a12293-4a77-4829-93da-7e1909f3853f\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") "
Nov 03 20:22:45.703797 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.703763    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-istio-envoy\") pod \"10a12293-4a77-4829-93da-7e1909f3853f\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") "
Nov 03 20:22:45.703972 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.703893    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-istio-data" pod "10a12293-4a77-4829-93da-7e1909f3853f" (UID: "10a12293-4a77-4829-93da-7e1909f3853f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:45.703972 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.703904    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10a12293-4a77-4829-93da-7e1909f3853f-istio-token\" (UniqueName: \"kubernetes.io/projected/10a12293-4a77-4829-93da-7e1909f3853f-istio-token\") pod \"10a12293-4a77-4829-93da-7e1909f3853f\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") "
Nov 03 20:22:45.704067 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.703967    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10a12293-4a77-4829-93da-7e1909f3853f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10a12293-4a77-4829-93da-7e1909f3853f-istio-podinfo\") pod \"10a12293-4a77-4829-93da-7e1909f3853f\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") "
Nov 03 20:22:45.704856 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.704381    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-workload-certs" pod "10a12293-4a77-4829-93da-7e1909f3853f" (UID: "10a12293-4a77-4829-93da-7e1909f3853f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:45.705020 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.704864    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-workload-socket" pod "10a12293-4a77-4829-93da-7e1909f3853f" (UID: "10a12293-4a77-4829-93da-7e1909f3853f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:45.705020 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.704949    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-credential-socket\") pod \"10a12293-4a77-4829-93da-7e1909f3853f\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") "
Nov 03 20:22:45.705020 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.704989    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10a12293-4a77-4829-93da-7e1909f3853f-kube-api-access-x8wxr\" (UniqueName: \"kubernetes.io/projected/10a12293-4a77-4829-93da-7e1909f3853f-kube-api-access-x8wxr\") pod \"10a12293-4a77-4829-93da-7e1909f3853f\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") "
Nov 03 20:22:45.705179 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.705097    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10a12293-4a77-4829-93da-7e1909f3853f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10a12293-4a77-4829-93da-7e1909f3853f-istiod-ca-cert\") pod \"10a12293-4a77-4829-93da-7e1909f3853f\" (UID: \"10a12293-4a77-4829-93da-7e1909f3853f\") "
Nov 03 20:22:45.705890 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.705868    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-credential-socket" pod "10a12293-4a77-4829-93da-7e1909f3853f" (UID: "10a12293-4a77-4829-93da-7e1909f3853f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:45.706078 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.706060    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10a12293-4a77-4829-93da-7e1909f3853f-istiod-ca-cert" pod "10a12293-4a77-4829-93da-7e1909f3853f" (UID: "10a12293-4a77-4829-93da-7e1909f3853f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:22:45.706190 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.706175    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:45.706252 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.706209    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:45.706252 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.706228    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:45.706937 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.706909    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a12293-4a77-4829-93da-7e1909f3853f-istio-token" pod "10a12293-4a77-4829-93da-7e1909f3853f" (UID: "10a12293-4a77-4829-93da-7e1909f3853f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:45.708052 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.708018    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-istio-envoy" pod "10a12293-4a77-4829-93da-7e1909f3853f" (UID: "10a12293-4a77-4829-93da-7e1909f3853f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:45.708458 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.708441    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10a12293-4a77-4829-93da-7e1909f3853f-istio-podinfo" pod "10a12293-4a77-4829-93da-7e1909f3853f" (UID: "10a12293-4a77-4829-93da-7e1909f3853f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:22:45.709776 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.709743    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a12293-4a77-4829-93da-7e1909f3853f-kube-api-access-x8wxr" pod "10a12293-4a77-4829-93da-7e1909f3853f" (UID: "10a12293-4a77-4829-93da-7e1909f3853f"). InnerVolumeSpecName "kube-api-access-x8wxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:45.808193 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.808103    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:45.808193 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.808135    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8wxr\" (UniqueName: \"kubernetes.io/projected/10a12293-4a77-4829-93da-7e1909f3853f-kube-api-access-x8wxr\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:45.808193 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.808145    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10a12293-4a77-4829-93da-7e1909f3853f-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:45.808193 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.808154    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10a12293-4a77-4829-93da-7e1909f3853f-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:45.808193 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.808162    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10a12293-4a77-4829-93da-7e1909f3853f-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:45.808193 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.808169    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10a12293-4a77-4829-93da-7e1909f3853f-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:45.999067 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.999024    4421 generic.go:356] "Generic (PLEG): container finished" podID="10a12293-4a77-4829-93da-7e1909f3853f" containerID="a9c524e56122a534d3b85a6391912e7e74d6bdf7af3c6b0a6fae7f752170c8ed" exitCode=0
Nov 03 20:22:45.999266 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.999111    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh" event={"ID":"10a12293-4a77-4829-93da-7e1909f3853f","Type":"ContainerDied","Data":"a9c524e56122a534d3b85a6391912e7e74d6bdf7af3c6b0a6fae7f752170c8ed"}
Nov 03 20:22:45.999266 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.999174    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh" event={"ID":"10a12293-4a77-4829-93da-7e1909f3853f","Type":"ContainerDied","Data":"ed1984d7ee9aaf93e1844b248c934a8471e250d05b14b1866df5474efe57cb54"}
Nov 03 20:22:45.999266 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.999197    4421 scope.go:122] "RemoveContainer" containerID="a9c524e56122a534d3b85a6391912e7e74d6bdf7af3c6b0a6fae7f752170c8ed"
Nov 03 20:22:45.999472 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:45.999452    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"
Nov 03 20:22:46.011896 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:46.011856    4421 scope.go:122] "RemoveContainer" containerID="c37b2c431f861865fb9b00217e1225156c455c23ff225fa7331b785bb7cbcaf1"
Nov 03 20:22:46.024239 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:46.024188    4421 scope.go:122] "RemoveContainer" containerID="a9c524e56122a534d3b85a6391912e7e74d6bdf7af3c6b0a6fae7f752170c8ed"
Nov 03 20:22:46.027005 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:22:46.025224    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9c524e56122a534d3b85a6391912e7e74d6bdf7af3c6b0a6fae7f752170c8ed\": container with ID starting with a9c524e56122a534d3b85a6391912e7e74d6bdf7af3c6b0a6fae7f752170c8ed not found: ID does not exist" containerID="a9c524e56122a534d3b85a6391912e7e74d6bdf7af3c6b0a6fae7f752170c8ed"
Nov 03 20:22:46.027005 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:46.025263    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9c524e56122a534d3b85a6391912e7e74d6bdf7af3c6b0a6fae7f752170c8ed"} err="failed to get container status \"a9c524e56122a534d3b85a6391912e7e74d6bdf7af3c6b0a6fae7f752170c8ed\": rpc error: code = NotFound desc = could not find container \"a9c524e56122a534d3b85a6391912e7e74d6bdf7af3c6b0a6fae7f752170c8ed\": container with ID starting with a9c524e56122a534d3b85a6391912e7e74d6bdf7af3c6b0a6fae7f752170c8ed not found: ID does not exist"
Nov 03 20:22:46.027005 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:46.025290    4421 scope.go:122] "RemoveContainer" containerID="c37b2c431f861865fb9b00217e1225156c455c23ff225fa7331b785bb7cbcaf1"
Nov 03 20:22:46.027005 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:22:46.025772    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c37b2c431f861865fb9b00217e1225156c455c23ff225fa7331b785bb7cbcaf1\": container with ID starting with c37b2c431f861865fb9b00217e1225156c455c23ff225fa7331b785bb7cbcaf1 not found: ID does not exist" containerID="c37b2c431f861865fb9b00217e1225156c455c23ff225fa7331b785bb7cbcaf1"
Nov 03 20:22:46.027005 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:46.025812    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37b2c431f861865fb9b00217e1225156c455c23ff225fa7331b785bb7cbcaf1"} err="failed to get container status \"c37b2c431f861865fb9b00217e1225156c455c23ff225fa7331b785bb7cbcaf1\": rpc error: code = NotFound desc = could not find container \"c37b2c431f861865fb9b00217e1225156c455c23ff225fa7331b785bb7cbcaf1\": container with ID starting with c37b2c431f861865fb9b00217e1225156c455c23ff225fa7331b785bb7cbcaf1 not found: ID does not exist"
Nov 03 20:22:46.034110 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:46.034085    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"]
Nov 03 20:22:46.038990 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:46.038967    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b07e26-openshift-default-78c99c7c89-gdxzh"]
Nov 03 20:22:46.961207 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:46.961174    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn"
Nov 03 20:22:47.073640 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.073587    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k"]
Nov 03 20:22:47.074307 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.074293    4421 cpu_manager.go:414] "containerMap: removing container" podUID="10a12293-4a77-4829-93da-7e1909f3853f" containerName="istio-proxy"
Nov 03 20:22:47.074361 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.074309    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10a12293-4a77-4829-93da-7e1909f3853f" containerName="istio-proxy"
Nov 03 20:22:47.074454 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.074444    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="10a12293-4a77-4829-93da-7e1909f3853f" containerName="istio-proxy"
Nov 03 20:22:47.079409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.079387    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k"
Nov 03 20:22:47.088263 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.088224    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-a596f7\""
Nov 03 20:22:47.095235 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.092756    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k"]
Nov 03 20:22:47.098049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.098025    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f"]
Nov 03 20:22:47.099090 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.099068    4421 cpu_manager.go:414] "containerMap: removing container" podUID="10a12293-4a77-4829-93da-7e1909f3853f" containerName="istio-proxy"
Nov 03 20:22:47.099179 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.099092    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10a12293-4a77-4829-93da-7e1909f3853f" containerName="istio-proxy"
Nov 03 20:22:47.099294 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.099282    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="10a12293-4a77-4829-93da-7e1909f3853f" containerName="istio-proxy"
Nov 03 20:22:47.103641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.103621    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f"
Nov 03 20:22:47.115667 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.111732    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-a596f7-dockercfg-zl8zt\""
Nov 03 20:22:47.122395 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.122373    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f"]
Nov 03 20:22:47.132262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.132234    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a53d3e21-8095-4e09-a80d-1b00732ea5b2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k\" (UID: \"a53d3e21-8095-4e09-a80d-1b00732ea5b2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k"
Nov 03 20:22:47.132366 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.132275    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svdbl\" (UniqueName: \"kubernetes.io/projected/a53d3e21-8095-4e09-a80d-1b00732ea5b2-kube-api-access-svdbl\") pod \"payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k\" (UID: \"a53d3e21-8095-4e09-a80d-1b00732ea5b2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k"
Nov 03 20:22:47.132366 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.132297    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjl6v\" (UniqueName: \"kubernetes.io/projected/4103ec14-a57a-465d-b80b-0a76abe1870e-kube-api-access-zjl6v\") pod \"payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f\" (UID: \"4103ec14-a57a-465d-b80b-0a76abe1870e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f"
Nov 03 20:22:47.132480 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.132469    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4103ec14-a57a-465d-b80b-0a76abe1870e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f\" (UID: \"4103ec14-a57a-465d-b80b-0a76abe1870e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f"
Nov 03 20:22:47.236851 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.233789    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a53d3e21-8095-4e09-a80d-1b00732ea5b2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k\" (UID: \"a53d3e21-8095-4e09-a80d-1b00732ea5b2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k"
Nov 03 20:22:47.236851 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.233860    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svdbl\" (UniqueName: \"kubernetes.io/projected/a53d3e21-8095-4e09-a80d-1b00732ea5b2-kube-api-access-svdbl\") pod \"payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k\" (UID: \"a53d3e21-8095-4e09-a80d-1b00732ea5b2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k"
Nov 03 20:22:47.236851 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.233897    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjl6v\" (UniqueName: \"kubernetes.io/projected/4103ec14-a57a-465d-b80b-0a76abe1870e-kube-api-access-zjl6v\") pod \"payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f\" (UID: \"4103ec14-a57a-465d-b80b-0a76abe1870e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f"
Nov 03 20:22:47.236851 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.234124    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4103ec14-a57a-465d-b80b-0a76abe1870e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f\" (UID: \"4103ec14-a57a-465d-b80b-0a76abe1870e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f"
Nov 03 20:22:47.236851 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.235251    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4103ec14-a57a-465d-b80b-0a76abe1870e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f\" (UID: \"4103ec14-a57a-465d-b80b-0a76abe1870e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f"
Nov 03 20:22:47.236851 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.236471    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a53d3e21-8095-4e09-a80d-1b00732ea5b2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k\" (UID: \"a53d3e21-8095-4e09-a80d-1b00732ea5b2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k"
Nov 03 20:22:47.288655 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.283139    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjl6v\" (UniqueName: \"kubernetes.io/projected/4103ec14-a57a-465d-b80b-0a76abe1870e-kube-api-access-zjl6v\") pod \"payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f\" (UID: \"4103ec14-a57a-465d-b80b-0a76abe1870e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f"
Nov 03 20:22:47.296621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.292773    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svdbl\" (UniqueName: \"kubernetes.io/projected/a53d3e21-8095-4e09-a80d-1b00732ea5b2-kube-api-access-svdbl\") pod \"payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k\" (UID: \"a53d3e21-8095-4e09-a80d-1b00732ea5b2\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k"
Nov 03 20:22:47.400622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.399865    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k"
Nov 03 20:22:47.437263 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.426241    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f"
Nov 03 20:22:47.781451 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.781418    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f"]
Nov 03 20:22:47.782539 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:22:47.782510    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4103ec14_a57a_465d_b80b_0a76abe1870e.slice/crio-1d007e31f77f0b9545913e17c13718330e29f72c9f28e13460b06146b922e7a0 WatchSource:0}: Error finding container 1d007e31f77f0b9545913e17c13718330e29f72c9f28e13460b06146b922e7a0: Status 404 returned error can't find the container with id 1d007e31f77f0b9545913e17c13718330e29f72c9f28e13460b06146b922e7a0
Nov 03 20:22:47.784328 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:22:47.784306    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53d3e21_8095_4e09_a80d_1b00732ea5b2.slice/crio-15ac737478e79a68456c359b1636ed5c4ff30e825655acf6b5127adc9eeda3d3 WatchSource:0}: Error finding container 15ac737478e79a68456c359b1636ed5c4ff30e825655acf6b5127adc9eeda3d3: Status 404 returned error can't find the container with id 15ac737478e79a68456c359b1636ed5c4ff30e825655acf6b5127adc9eeda3d3
Nov 03 20:22:47.785923 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:47.785794    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k"]
Nov 03 20:22:48.094289 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:48.094172    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10a12293-4a77-4829-93da-7e1909f3853f" path="/var/lib/kubelet/pods/10a12293-4a77-4829-93da-7e1909f3853f/volumes"
Nov 03 20:22:48.094928 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:48.094694    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f" event={"ID":"4103ec14-a57a-465d-b80b-0a76abe1870e","Type":"ContainerStarted","Data":"a5cd412b183e23a469b158ba7ec672aae4ecd85830959862f6f7aa59f44bd67a"}
Nov 03 20:22:48.094928 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:48.094754    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f" event={"ID":"4103ec14-a57a-465d-b80b-0a76abe1870e","Type":"ContainerStarted","Data":"1d007e31f77f0b9545913e17c13718330e29f72c9f28e13460b06146b922e7a0"}
Nov 03 20:22:48.094928 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:48.094782    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k" event={"ID":"a53d3e21-8095-4e09-a80d-1b00732ea5b2","Type":"ContainerStarted","Data":"03263ae82c60546392a7cbc7ba1950cfe54eacaf41c455e3e76057896a87a5f2"}
Nov 03 20:22:48.094928 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:48.094813    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f"
Nov 03 20:22:48.094928 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:48.094834    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k" event={"ID":"a53d3e21-8095-4e09-a80d-1b00732ea5b2","Type":"ContainerStarted","Data":"15ac737478e79a68456c359b1636ed5c4ff30e825655acf6b5127adc9eeda3d3"}
Nov 03 20:22:48.094928 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:48.094858    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k"
Nov 03 20:22:48.112162 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:48.112120    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f" podStartSLOduration=1.112106455 podStartE2EDuration="1.112106455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:48.109880979 +0000 UTC m=+1298.243009441" watchObservedRunningTime="2026-11-03 20:22:48.112106455 +0000 UTC m=+1298.245234944"
Nov 03 20:22:48.132873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:48.132825    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k" podStartSLOduration=1.132811227 podStartE2EDuration="1.132811227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:48.12811113 +0000 UTC m=+1298.261239592" watchObservedRunningTime="2026-11-03 20:22:48.132811227 +0000 UTC m=+1298.265939732"
Nov 03 20:22:48.978648 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:48.970532    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp"
Nov 03 20:22:48.978648 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:48.975843    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm"
Nov 03 20:22:49.348641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.346027    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:22:49.387914 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.387883    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:22:49.468759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.463879    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-podinfo\") pod \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") "
Nov 03 20:22:49.468759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.463963    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-kube-api-access-kpwfn\" (UniqueName: \"kubernetes.io/projected/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-kube-api-access-kpwfn\") pod \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") "
Nov 03 20:22:49.468759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.464022    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-workload-socket\") pod \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") "
Nov 03 20:22:49.468759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.464055    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-credential-socket\") pod \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") "
Nov 03 20:22:49.468759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.464103    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34c80431-a853-4012-b239-ea4659c70bf9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c80431-a853-4012-b239-ea4659c70bf9-istio-podinfo\") pod \"34c80431-a853-4012-b239-ea4659c70bf9\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") "
Nov 03 20:22:49.468759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.464153    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34c80431-a853-4012-b239-ea4659c70bf9-istio-token\" (UniqueName: \"kubernetes.io/projected/34c80431-a853-4012-b239-ea4659c70bf9-istio-token\") pod \"34c80431-a853-4012-b239-ea4659c70bf9\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") "
Nov 03 20:22:49.468759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.464189    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34c80431-a853-4012-b239-ea4659c70bf9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c80431-a853-4012-b239-ea4659c70bf9-istiod-ca-cert\") pod \"34c80431-a853-4012-b239-ea4659c70bf9\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") "
Nov 03 20:22:49.468759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.464250    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-workload-certs\") pod \"34c80431-a853-4012-b239-ea4659c70bf9\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") "
Nov 03 20:22:49.468759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.464292    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34c80431-a853-4012-b239-ea4659c70bf9-kube-api-access-hfrjx\" (UniqueName: \"kubernetes.io/projected/34c80431-a853-4012-b239-ea4659c70bf9-kube-api-access-hfrjx\") pod \"34c80431-a853-4012-b239-ea4659c70bf9\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") "
Nov 03 20:22:49.468759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.464350    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-credential-socket\") pod \"34c80431-a853-4012-b239-ea4659c70bf9\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") "
Nov 03 20:22:49.468759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.464405    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-istio-envoy\") pod \"34c80431-a853-4012-b239-ea4659c70bf9\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") "
Nov 03 20:22:49.468759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.464464    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-token\" (UniqueName: \"kubernetes.io/projected/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-token\") pod \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") "
Nov 03 20:22:49.468759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.465067    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-data\") pod \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") "
Nov 03 20:22:49.468759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.465127    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-workload-socket\") pod \"34c80431-a853-4012-b239-ea4659c70bf9\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") "
Nov 03 20:22:49.468759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.465295    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-workload-certs\") pod \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") "
Nov 03 20:22:49.468759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.465358    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-envoy\") pod \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") "
Nov 03 20:22:49.468759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.465406    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istiod-ca-cert\") pod \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\" (UID: \"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc\") "
Nov 03 20:22:49.468759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.465464    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-istio-data\") pod \"34c80431-a853-4012-b239-ea4659c70bf9\" (UID: \"34c80431-a853-4012-b239-ea4659c70bf9\") "
Nov 03 20:22:49.468759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.466785    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-workload-certs" pod "08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc" (UID: "08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:49.468759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.466975    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-data" pod "08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc" (UID: "08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:49.468759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.467150    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-workload-socket" pod "34c80431-a853-4012-b239-ea4659c70bf9" (UID: "34c80431-a853-4012-b239-ea4659c70bf9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:49.468759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.468034    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-credential-socket" pod "34c80431-a853-4012-b239-ea4659c70bf9" (UID: "34c80431-a853-4012-b239-ea4659c70bf9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:49.468759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.468325    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34c80431-a853-4012-b239-ea4659c70bf9-istiod-ca-cert" pod "34c80431-a853-4012-b239-ea4659c70bf9" (UID: "34c80431-a853-4012-b239-ea4659c70bf9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:22:49.470026 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.468965    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-workload-socket" pod "08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc" (UID: "08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:49.470026 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.469652    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istiod-ca-cert" pod "08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc" (UID: "08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:22:49.472111 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.472079    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-workload-certs" pod "34c80431-a853-4012-b239-ea4659c70bf9" (UID: "34c80431-a853-4012-b239-ea4659c70bf9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:49.472289 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.472266    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-credential-socket" pod "08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc" (UID: "08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:49.474052 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.474022    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-istio-envoy" pod "34c80431-a853-4012-b239-ea4659c70bf9" (UID: "34c80431-a853-4012-b239-ea4659c70bf9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:49.474164 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.474138    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-token" pod "08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc" (UID: "08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:49.475044 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.475017    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c80431-a853-4012-b239-ea4659c70bf9-kube-api-access-hfrjx" pod "34c80431-a853-4012-b239-ea4659c70bf9" (UID: "34c80431-a853-4012-b239-ea4659c70bf9"). InnerVolumeSpecName "kube-api-access-hfrjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:49.476081 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.476054    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-istio-data" pod "34c80431-a853-4012-b239-ea4659c70bf9" (UID: "34c80431-a853-4012-b239-ea4659c70bf9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:49.476531 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.476506    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-kube-api-access-kpwfn" pod "08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc" (UID: "08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc"). InnerVolumeSpecName "kube-api-access-kpwfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:49.477427 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.477392    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34c80431-a853-4012-b239-ea4659c70bf9-istio-podinfo" pod "34c80431-a853-4012-b239-ea4659c70bf9" (UID: "34c80431-a853-4012-b239-ea4659c70bf9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:22:49.486060 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.486004    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-envoy" pod "08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc" (UID: "08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:49.486204 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.486178    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c80431-a853-4012-b239-ea4659c70bf9-istio-token" pod "34c80431-a853-4012-b239-ea4659c70bf9" (UID: "34c80431-a853-4012-b239-ea4659c70bf9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:49.488052 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.488025    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-podinfo" pod "08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc" (UID: "08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:22:49.567916 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.567872    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:49.567916 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.567906    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:49.567916 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.567920    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:49.568219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.567934    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:49.568219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.567952    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:49.568219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.567964    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:49.568219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.567980    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:49.568219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.567993    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:49.568219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.568008    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:49.568219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.568021    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpwfn\" (UniqueName: \"kubernetes.io/projected/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-kube-api-access-kpwfn\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:49.568219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.568036    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:49.568219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.568048    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:49.568219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.568061    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c80431-a853-4012-b239-ea4659c70bf9-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:49.568219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.568075    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c80431-a853-4012-b239-ea4659c70bf9-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:49.568219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.568088    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c80431-a853-4012-b239-ea4659c70bf9-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:49.568219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.568101    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:49.568219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.568117    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfrjx\" (UniqueName: \"kubernetes.io/projected/34c80431-a853-4012-b239-ea4659c70bf9-kube-api-access-hfrjx\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:49.568219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.568130    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c80431-a853-4012-b239-ea4659c70bf9-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:49.630659 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:22:49.630576    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:22:49.941932 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.941898    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6"
Nov 03 20:22:49.942107 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:49.941977    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj"
Nov 03 20:22:50.116088 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.116049    4421 generic.go:356] "Generic (PLEG): container finished" podID="08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc" containerID="bde8000b904e5f5f581abdb127e2a365a8b36201d2e4dfc6815a921bb8d05cfe" exitCode=0
Nov 03 20:22:50.116257 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.116172    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm" event={"ID":"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc","Type":"ContainerDied","Data":"bde8000b904e5f5f581abdb127e2a365a8b36201d2e4dfc6815a921bb8d05cfe"}
Nov 03 20:22:50.116257 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.116221    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm" event={"ID":"08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc","Type":"ContainerDied","Data":"066ab2105adc1581c6a3816bc5066fca3916efe6ce3a1be44302db7ebd201637"}
Nov 03 20:22:50.116257 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.116247    4421 scope.go:122] "RemoveContainer" containerID="bde8000b904e5f5f581abdb127e2a365a8b36201d2e4dfc6815a921bb8d05cfe"
Nov 03 20:22:50.116438 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.116377    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"
Nov 03 20:22:50.123884 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.123853    4421 generic.go:356] "Generic (PLEG): container finished" podID="34c80431-a853-4012-b239-ea4659c70bf9" containerID="84c1151db91f57fbd6758c31846f3666bc635f87815afe117e56eac0220901db" exitCode=0
Nov 03 20:22:50.124024 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.123971    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z" event={"ID":"34c80431-a853-4012-b239-ea4659c70bf9","Type":"ContainerDied","Data":"84c1151db91f57fbd6758c31846f3666bc635f87815afe117e56eac0220901db"}
Nov 03 20:22:50.124024 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.124018    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z" event={"ID":"34c80431-a853-4012-b239-ea4659c70bf9","Type":"ContainerDied","Data":"86c686a33637de6ca7b42b2f782a7ee085cb8b1c36842cefbe04648d8dbc099e"}
Nov 03 20:22:50.128045 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.128028    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"
Nov 03 20:22:50.137309 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.137286    4421 scope.go:122] "RemoveContainer" containerID="bde8000b904e5f5f581abdb127e2a365a8b36201d2e4dfc6815a921bb8d05cfe"
Nov 03 20:22:50.137704 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:22:50.137675    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde8000b904e5f5f581abdb127e2a365a8b36201d2e4dfc6815a921bb8d05cfe\": container with ID starting with bde8000b904e5f5f581abdb127e2a365a8b36201d2e4dfc6815a921bb8d05cfe not found: ID does not exist" containerID="bde8000b904e5f5f581abdb127e2a365a8b36201d2e4dfc6815a921bb8d05cfe"
Nov 03 20:22:50.137816 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.137713    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde8000b904e5f5f581abdb127e2a365a8b36201d2e4dfc6815a921bb8d05cfe"} err="failed to get container status \"bde8000b904e5f5f581abdb127e2a365a8b36201d2e4dfc6815a921bb8d05cfe\": rpc error: code = NotFound desc = could not find container \"bde8000b904e5f5f581abdb127e2a365a8b36201d2e4dfc6815a921bb8d05cfe\": container with ID starting with bde8000b904e5f5f581abdb127e2a365a8b36201d2e4dfc6815a921bb8d05cfe not found: ID does not exist"
Nov 03 20:22:50.137816 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.137736    4421 scope.go:122] "RemoveContainer" containerID="84c1151db91f57fbd6758c31846f3666bc635f87815afe117e56eac0220901db"
Nov 03 20:22:50.225149 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.221325    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"]
Nov 03 20:22:50.225149 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.221389    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-4mlxm"]
Nov 03 20:22:50.248671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.248648    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"]
Nov 03 20:22:50.260888 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.257177    4421 scope.go:122] "RemoveContainer" containerID="84c1151db91f57fbd6758c31846f3666bc635f87815afe117e56eac0220901db"
Nov 03 20:22:50.260888 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:22:50.257561    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84c1151db91f57fbd6758c31846f3666bc635f87815afe117e56eac0220901db\": container with ID starting with 84c1151db91f57fbd6758c31846f3666bc635f87815afe117e56eac0220901db not found: ID does not exist" containerID="84c1151db91f57fbd6758c31846f3666bc635f87815afe117e56eac0220901db"
Nov 03 20:22:50.260888 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.257658    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84c1151db91f57fbd6758c31846f3666bc635f87815afe117e56eac0220901db"} err="failed to get container status \"84c1151db91f57fbd6758c31846f3666bc635f87815afe117e56eac0220901db\": rpc error: code = NotFound desc = could not find container \"84c1151db91f57fbd6758c31846f3666bc635f87815afe117e56eac0220901db\": container with ID starting with 84c1151db91f57fbd6758c31846f3666bc635f87815afe117e56eac0220901db not found: ID does not exist"
Nov 03 20:22:50.260888 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.258045    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-7j92z"]
Nov 03 20:22:50.495884 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.495859    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:22:50.500274 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.500248    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:22:50.599434 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.599401    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-data\") pod \"7bb10f7c-7a24-482b-961e-966a83c8139d\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") "
Nov 03 20:22:50.599631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.599451    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-envoy\") pod \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") "
Nov 03 20:22:50.599631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.599493    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-token\" (UniqueName: \"kubernetes.io/projected/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-token\") pod \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") "
Nov 03 20:22:50.599631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.599535    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-workload-certs\") pod \"7bb10f7c-7a24-482b-961e-966a83c8139d\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") "
Nov 03 20:22:50.599872 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.599635    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bb10f7c-7a24-482b-961e-966a83c8139d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bb10f7c-7a24-482b-961e-966a83c8139d-istiod-ca-cert\") pod \"7bb10f7c-7a24-482b-961e-966a83c8139d\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") "
Nov 03 20:22:50.599872 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.599709    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-data" pod "7bb10f7c-7a24-482b-961e-966a83c8139d" (UID: "7bb10f7c-7a24-482b-961e-966a83c8139d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:50.599872 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.599741    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-credential-socket\") pod \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") "
Nov 03 20:22:50.599872 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.599799    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-token\" (UniqueName: \"kubernetes.io/projected/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-token\") pod \"7bb10f7c-7a24-482b-961e-966a83c8139d\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") "
Nov 03 20:22:50.599872 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.599824    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-envoy\") pod \"7bb10f7c-7a24-482b-961e-966a83c8139d\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") "
Nov 03 20:22:50.599872 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.599867    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-kube-api-access-cv26v\" (UniqueName: \"kubernetes.io/projected/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-kube-api-access-cv26v\") pod \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") "
Nov 03 20:22:50.600421 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.599927    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istiod-ca-cert\") pod \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") "
Nov 03 20:22:50.600421 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.599927    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-workload-certs" pod "7bb10f7c-7a24-482b-961e-966a83c8139d" (UID: "7bb10f7c-7a24-482b-961e-966a83c8139d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:50.600421 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.599998    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bb10f7c-7a24-482b-961e-966a83c8139d-kube-api-access-kzp5h\" (UniqueName: \"kubernetes.io/projected/7bb10f7c-7a24-482b-961e-966a83c8139d-kube-api-access-kzp5h\") pod \"7bb10f7c-7a24-482b-961e-966a83c8139d\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") "
Nov 03 20:22:50.600421 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.600035    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-workload-certs\") pod \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") "
Nov 03 20:22:50.600421 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.600068    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-workload-socket\") pod \"7bb10f7c-7a24-482b-961e-966a83c8139d\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") "
Nov 03 20:22:50.600421 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.600109    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-credential-socket\") pod \"7bb10f7c-7a24-482b-961e-966a83c8139d\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") "
Nov 03 20:22:50.600421 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.600200    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-podinfo\") pod \"7bb10f7c-7a24-482b-961e-966a83c8139d\" (UID: \"7bb10f7c-7a24-482b-961e-966a83c8139d\") "
Nov 03 20:22:50.600421 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.600239    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-podinfo\") pod \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") "
Nov 03 20:22:50.600421 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.600269    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-workload-socket\") pod \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") "
Nov 03 20:22:50.600421 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.600315    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-data\") pod \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\" (UID: \"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c\") "
Nov 03 20:22:50.600961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.600823    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb10f7c-7a24-482b-961e-966a83c8139d-istiod-ca-cert" pod "7bb10f7c-7a24-482b-961e-966a83c8139d" (UID: "7bb10f7c-7a24-482b-961e-966a83c8139d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:22:50.601177 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.601157    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bb10f7c-7a24-482b-961e-966a83c8139d-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:50.601288 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.601266    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:50.601355 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.601292    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:50.601355 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.601330    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-data" pod "8f6f3683-cb02-4dd2-aed6-a919a84c2e4c" (UID: "8f6f3683-cb02-4dd2-aed6-a919a84c2e4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:50.601873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.601632    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-credential-socket" pod "8f6f3683-cb02-4dd2-aed6-a919a84c2e4c" (UID: "8f6f3683-cb02-4dd2-aed6-a919a84c2e4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:50.602256 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.602024    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-workload-certs" pod "8f6f3683-cb02-4dd2-aed6-a919a84c2e4c" (UID: "8f6f3683-cb02-4dd2-aed6-a919a84c2e4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:50.602744 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.602480    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istiod-ca-cert" pod "8f6f3683-cb02-4dd2-aed6-a919a84c2e4c" (UID: "8f6f3683-cb02-4dd2-aed6-a919a84c2e4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:22:50.605474 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.605108    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-podinfo" pod "8f6f3683-cb02-4dd2-aed6-a919a84c2e4c" (UID: "8f6f3683-cb02-4dd2-aed6-a919a84c2e4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:22:50.605474 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.605285    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-envoy" pod "8f6f3683-cb02-4dd2-aed6-a919a84c2e4c" (UID: "8f6f3683-cb02-4dd2-aed6-a919a84c2e4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:50.605474 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.605319    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-kube-api-access-cv26v" pod "8f6f3683-cb02-4dd2-aed6-a919a84c2e4c" (UID: "8f6f3683-cb02-4dd2-aed6-a919a84c2e4c"). InnerVolumeSpecName "kube-api-access-cv26v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:50.605711 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.605502    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-credential-socket" pod "7bb10f7c-7a24-482b-961e-966a83c8139d" (UID: "7bb10f7c-7a24-482b-961e-966a83c8139d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:50.605785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.605736    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-workload-socket" pod "8f6f3683-cb02-4dd2-aed6-a919a84c2e4c" (UID: "8f6f3683-cb02-4dd2-aed6-a919a84c2e4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:50.605910 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.605881    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-workload-socket" pod "7bb10f7c-7a24-482b-961e-966a83c8139d" (UID: "7bb10f7c-7a24-482b-961e-966a83c8139d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:50.606166 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.606148    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-token" pod "7bb10f7c-7a24-482b-961e-966a83c8139d" (UID: "7bb10f7c-7a24-482b-961e-966a83c8139d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:50.606920 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.606800    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-envoy" pod "7bb10f7c-7a24-482b-961e-966a83c8139d" (UID: "7bb10f7c-7a24-482b-961e-966a83c8139d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:50.606920 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.606893    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-token" pod "8f6f3683-cb02-4dd2-aed6-a919a84c2e4c" (UID: "8f6f3683-cb02-4dd2-aed6-a919a84c2e4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:50.606920 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.606897    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-podinfo" pod "7bb10f7c-7a24-482b-961e-966a83c8139d" (UID: "7bb10f7c-7a24-482b-961e-966a83c8139d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:22:50.607230 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.607204    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb10f7c-7a24-482b-961e-966a83c8139d-kube-api-access-kzp5h" pod "7bb10f7c-7a24-482b-961e-966a83c8139d" (UID: "7bb10f7c-7a24-482b-961e-966a83c8139d"). InnerVolumeSpecName "kube-api-access-kzp5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:50.702382 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.702343    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:50.702382 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.702372    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:50.702382 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.702381    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:50.702382 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.702390    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv26v\" (UniqueName: \"kubernetes.io/projected/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-kube-api-access-cv26v\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:50.702738 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.702398    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:50.702738 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.702409    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzp5h\" (UniqueName: \"kubernetes.io/projected/7bb10f7c-7a24-482b-961e-966a83c8139d-kube-api-access-kzp5h\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:50.702738 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.702417    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:50.702738 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.702426    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:50.702738 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.702434    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb10f7c-7a24-482b-961e-966a83c8139d-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:50.702738 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.702442    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bb10f7c-7a24-482b-961e-966a83c8139d-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:50.702738 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.702451    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:50.702738 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.702459    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:50.702738 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.702467    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:50.702738 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.702474    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:50.702738 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:50.702483    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:51.131963 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:51.131928    4421 generic.go:356] "Generic (PLEG): container finished" podID="8f6f3683-cb02-4dd2-aed6-a919a84c2e4c" containerID="49d100bfcb4c65564d1f8d4e077dcbac06ab79aa87b82ae2fa63d75be59ed94a" exitCode=0
Nov 03 20:22:51.132124 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:51.131998    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"
Nov 03 20:22:51.132124 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:51.132008    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb" event={"ID":"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c","Type":"ContainerDied","Data":"49d100bfcb4c65564d1f8d4e077dcbac06ab79aa87b82ae2fa63d75be59ed94a"}
Nov 03 20:22:51.132124 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:51.132084    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb" event={"ID":"8f6f3683-cb02-4dd2-aed6-a919a84c2e4c","Type":"ContainerDied","Data":"079618efe6c4c882a28e2b5fb271f831b0aa49d9e09df021162b6bc3c3241a7c"}
Nov 03 20:22:51.132124 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:51.132113    4421 scope.go:122] "RemoveContainer" containerID="49d100bfcb4c65564d1f8d4e077dcbac06ab79aa87b82ae2fa63d75be59ed94a"
Nov 03 20:22:51.133457 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:51.133423    4421 generic.go:356] "Generic (PLEG): container finished" podID="7bb10f7c-7a24-482b-961e-966a83c8139d" containerID="cc649c45aff40c543c3d7cb57f11823f915340bf656528282609bb229d9c3af0" exitCode=0
Nov 03 20:22:51.133593 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:51.133462    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5" event={"ID":"7bb10f7c-7a24-482b-961e-966a83c8139d","Type":"ContainerDied","Data":"cc649c45aff40c543c3d7cb57f11823f915340bf656528282609bb229d9c3af0"}
Nov 03 20:22:51.133593 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:51.133479    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"
Nov 03 20:22:51.133593 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:51.133517    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5" event={"ID":"7bb10f7c-7a24-482b-961e-966a83c8139d","Type":"ContainerDied","Data":"2ae5f460ef58ad83e7d88ad2398d50851cff6f2140fffaf9f3f3bb3c9320f298"}
Nov 03 20:22:51.145157 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:51.145137    4421 scope.go:122] "RemoveContainer" containerID="49d100bfcb4c65564d1f8d4e077dcbac06ab79aa87b82ae2fa63d75be59ed94a"
Nov 03 20:22:51.145415 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:22:51.145393    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49d100bfcb4c65564d1f8d4e077dcbac06ab79aa87b82ae2fa63d75be59ed94a\": container with ID starting with 49d100bfcb4c65564d1f8d4e077dcbac06ab79aa87b82ae2fa63d75be59ed94a not found: ID does not exist" containerID="49d100bfcb4c65564d1f8d4e077dcbac06ab79aa87b82ae2fa63d75be59ed94a"
Nov 03 20:22:51.145484 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:51.145426    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49d100bfcb4c65564d1f8d4e077dcbac06ab79aa87b82ae2fa63d75be59ed94a"} err="failed to get container status \"49d100bfcb4c65564d1f8d4e077dcbac06ab79aa87b82ae2fa63d75be59ed94a\": rpc error: code = NotFound desc = could not find container \"49d100bfcb4c65564d1f8d4e077dcbac06ab79aa87b82ae2fa63d75be59ed94a\": container with ID starting with 49d100bfcb4c65564d1f8d4e077dcbac06ab79aa87b82ae2fa63d75be59ed94a not found: ID does not exist"
Nov 03 20:22:51.145484 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:51.145446    4421 scope.go:122] "RemoveContainer" containerID="cc649c45aff40c543c3d7cb57f11823f915340bf656528282609bb229d9c3af0"
Nov 03 20:22:51.155965 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:51.155946    4421 scope.go:122] "RemoveContainer" containerID="cc649c45aff40c543c3d7cb57f11823f915340bf656528282609bb229d9c3af0"
Nov 03 20:22:51.156245 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:22:51.156225    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc649c45aff40c543c3d7cb57f11823f915340bf656528282609bb229d9c3af0\": container with ID starting with cc649c45aff40c543c3d7cb57f11823f915340bf656528282609bb229d9c3af0 not found: ID does not exist" containerID="cc649c45aff40c543c3d7cb57f11823f915340bf656528282609bb229d9c3af0"
Nov 03 20:22:51.156321 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:51.156256    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc649c45aff40c543c3d7cb57f11823f915340bf656528282609bb229d9c3af0"} err="failed to get container status \"cc649c45aff40c543c3d7cb57f11823f915340bf656528282609bb229d9c3af0\": rpc error: code = NotFound desc = could not find container \"cc649c45aff40c543c3d7cb57f11823f915340bf656528282609bb229d9c3af0\": container with ID starting with cc649c45aff40c543c3d7cb57f11823f915340bf656528282609bb229d9c3af0 not found: ID does not exist"
Nov 03 20:22:51.161306 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:51.161283    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"]
Nov 03 20:22:51.166347 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:51.166325    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-phzjb"]
Nov 03 20:22:51.178157 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:51.178139    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"]
Nov 03 20:22:51.182275 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:51.182257    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-77vt5"]
Nov 03 20:22:52.018615 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:52.018563    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc" path="/var/lib/kubelet/pods/08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc/volumes"
Nov 03 20:22:52.019259 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:52.019099    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34c80431-a853-4012-b239-ea4659c70bf9" path="/var/lib/kubelet/pods/34c80431-a853-4012-b239-ea4659c70bf9/volumes"
Nov 03 20:22:52.019509 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:52.019494    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bb10f7c-7a24-482b-961e-966a83c8139d" path="/var/lib/kubelet/pods/7bb10f7c-7a24-482b-961e-966a83c8139d/volumes"
Nov 03 20:22:52.020292 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:52.020276    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f6f3683-cb02-4dd2-aed6-a919a84c2e4c" path="/var/lib/kubelet/pods/8f6f3683-cb02-4dd2-aed6-a919a84c2e4c/volumes"
Nov 03 20:22:56.816442 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:56.816402    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8"]
Nov 03 20:22:56.817525 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:56.817498    4421 cpu_manager.go:414] "containerMap: removing container" podUID="8f6f3683-cb02-4dd2-aed6-a919a84c2e4c" containerName="istio-proxy"
Nov 03 20:22:56.817739 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:56.817715    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f6f3683-cb02-4dd2-aed6-a919a84c2e4c" containerName="istio-proxy"
Nov 03 20:22:56.817876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:56.817769    4421 cpu_manager.go:414] "containerMap: removing container" podUID="08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc" containerName="istio-proxy"
Nov 03 20:22:56.817876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:56.817780    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc" containerName="istio-proxy"
Nov 03 20:22:56.817876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:56.817817    4421 cpu_manager.go:414] "containerMap: removing container" podUID="34c80431-a853-4012-b239-ea4659c70bf9" containerName="istio-proxy"
Nov 03 20:22:56.817876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:56.817825    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34c80431-a853-4012-b239-ea4659c70bf9" containerName="istio-proxy"
Nov 03 20:22:56.817876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:56.817841    4421 cpu_manager.go:414] "containerMap: removing container" podUID="7bb10f7c-7a24-482b-961e-966a83c8139d" containerName="istio-proxy"
Nov 03 20:22:56.817876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:56.817848    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bb10f7c-7a24-482b-961e-966a83c8139d" containerName="istio-proxy"
Nov 03 20:22:56.818838 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:56.818052    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="34c80431-a853-4012-b239-ea4659c70bf9" containerName="istio-proxy"
Nov 03 20:22:56.818838 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:56.818068    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bb10f7c-7a24-482b-961e-966a83c8139d" containerName="istio-proxy"
Nov 03 20:22:56.818838 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:56.818087    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f6f3683-cb02-4dd2-aed6-a919a84c2e4c" containerName="istio-proxy"
Nov 03 20:22:56.818838 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:56.818098    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="08eb9bd3-5d1e-4284-bd0e-e3fb8f10adfc" containerName="istio-proxy"
Nov 03 20:22:56.824199 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:56.824153    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8"
Nov 03 20:22:56.835059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:56.833243    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8"]
Nov 03 20:22:56.835536 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:56.835255    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-a596f7-serving-cert\""
Nov 03 20:22:56.915699 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:56.915659    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25115ccd-00e2-48d2-8202-654e4920800f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8\" (UID: \"25115ccd-00e2-48d2-8202-654e4920800f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8"
Nov 03 20:22:56.915699 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:56.915701    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j776s\" (UniqueName: \"kubernetes.io/projected/25115ccd-00e2-48d2-8202-654e4920800f-kube-api-access-j776s\") pod \"maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8\" (UID: \"25115ccd-00e2-48d2-8202-654e4920800f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8"
Nov 03 20:22:56.915961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:56.915743    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25115ccd-00e2-48d2-8202-654e4920800f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8\" (UID: \"25115ccd-00e2-48d2-8202-654e4920800f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8"
Nov 03 20:22:56.915961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:56.915794    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25115ccd-00e2-48d2-8202-654e4920800f-metrics-certs\") pod \"maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8\" (UID: \"25115ccd-00e2-48d2-8202-654e4920800f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8"
Nov 03 20:22:57.017622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:57.017311    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25115ccd-00e2-48d2-8202-654e4920800f-metrics-certs\") pod \"maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8\" (UID: \"25115ccd-00e2-48d2-8202-654e4920800f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8"
Nov 03 20:22:57.017818 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:57.017784    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25115ccd-00e2-48d2-8202-654e4920800f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8\" (UID: \"25115ccd-00e2-48d2-8202-654e4920800f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8"
Nov 03 20:22:57.017818 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:57.017814    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j776s\" (UniqueName: \"kubernetes.io/projected/25115ccd-00e2-48d2-8202-654e4920800f-kube-api-access-j776s\") pod \"maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8\" (UID: \"25115ccd-00e2-48d2-8202-654e4920800f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8"
Nov 03 20:22:57.017971 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:57.017871    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25115ccd-00e2-48d2-8202-654e4920800f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8\" (UID: \"25115ccd-00e2-48d2-8202-654e4920800f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8"
Nov 03 20:22:57.027662 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:57.019145    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25115ccd-00e2-48d2-8202-654e4920800f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8\" (UID: \"25115ccd-00e2-48d2-8202-654e4920800f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8"
Nov 03 20:22:57.046622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:57.043717    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25115ccd-00e2-48d2-8202-654e4920800f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8\" (UID: \"25115ccd-00e2-48d2-8202-654e4920800f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8"
Nov 03 20:22:57.065796 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:57.054329    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j776s\" (UniqueName: \"kubernetes.io/projected/25115ccd-00e2-48d2-8202-654e4920800f-kube-api-access-j776s\") pod \"maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8\" (UID: \"25115ccd-00e2-48d2-8202-654e4920800f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8"
Nov 03 20:22:57.073676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:57.067213    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25115ccd-00e2-48d2-8202-654e4920800f-metrics-certs\") pod \"maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8\" (UID: \"25115ccd-00e2-48d2-8202-654e4920800f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8"
Nov 03 20:22:57.163622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:57.161892    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8"
Nov 03 20:22:57.478838 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:57.478802    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8"]
Nov 03 20:22:57.479823 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:22:57.479770    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25115ccd_00e2_48d2_8202_654e4920800f.slice/crio-e93dd075405392ffdf52d1b9441b73832866fd5db4ff893a59830fc3eb9b1c41 WatchSource:0}: Error finding container e93dd075405392ffdf52d1b9441b73832866fd5db4ff893a59830fc3eb9b1c41: Status 404 returned error can't find the container with id e93dd075405392ffdf52d1b9441b73832866fd5db4ff893a59830fc3eb9b1c41
Nov 03 20:22:57.481531 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:57.481514    4421 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:22:58.271843 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.271747    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8" event={"ID":"25115ccd-00e2-48d2-8202-654e4920800f","Type":"ContainerStarted","Data":"6d9be67eee6cce135a4fca1fb2842903b3b418de822f6ba20695c77db088facd"}
Nov 03 20:22:58.272408 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.272382    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8" event={"ID":"25115ccd-00e2-48d2-8202-654e4920800f","Type":"ContainerStarted","Data":"e93dd075405392ffdf52d1b9441b73832866fd5db4ff893a59830fc3eb9b1c41"}
Nov 03 20:22:58.274830 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.273577    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8"
Nov 03 20:22:58.312624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.310065    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8" podStartSLOduration=1.814513128 podStartE2EDuration="2.310040929s" totalImagesPullingTime="495.527801ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:58.297193052 +0000 UTC m=+1308.430321575" watchObservedRunningTime="2026-11-03 20:22:58.310040929 +0000 UTC m=+1308.443169392"
Nov 03 20:22:58.483654 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.483613    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"]
Nov 03 20:22:58.491789 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.491750    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.497100 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.496884    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-c4f356-openshift-default-dockercfg-j5x9r\""
Nov 03 20:22:58.520479 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.520450    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"]
Nov 03 20:22:58.630672 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.628926    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-podinfo\") pod \"e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") " pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.630672 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.628986    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-workload-socket\") pod \"e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") " pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.630672 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.629052    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istiod-ca-cert\") pod \"e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") " pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.630672 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.629107    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-workload-certs\") pod \"e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") " pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.630672 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.629137    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-data\") pod \"e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") " pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.630672 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.629233    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-token\") pod \"e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") " pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.630672 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.629265    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-envoy\") pod \"e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") " pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.630672 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.629325    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-credential-socket\") pod \"e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") " pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.630672 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.629436    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg5vk\" (UniqueName: \"kubernetes.io/projected/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-kube-api-access-wg5vk\") pod \"e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") " pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.737636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.732756    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-token\") pod \"e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") " pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.737636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.732810    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-envoy\") pod \"e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") " pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.737636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.732863    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-credential-socket\") pod \"e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") " pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.737636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.732962    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg5vk\" (UniqueName: \"kubernetes.io/projected/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-kube-api-access-wg5vk\") pod \"e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") " pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.737636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.733102    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-podinfo\") pod \"e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") " pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.737636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.733144    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-workload-socket\") pod \"e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") " pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.737636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.733205    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istiod-ca-cert\") pod \"e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") " pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.737636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.733259    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-workload-certs\") pod \"e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") " pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.737636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.733295    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-data\") pod \"e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") " pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.737636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.733842    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-data\") pod \"e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") " pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.755296 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.753577    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istiod-ca-cert\") pod \"e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") " pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.755296 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.753870    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-workload-socket\") pod \"e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") " pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.755296 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.754101    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-workload-certs\") pod \"e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") " pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.755296 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.754351    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-credential-socket\") pod \"e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") " pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.761899 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.761874    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-podinfo\") pod \"e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") " pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.797779 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.797707    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-token\") pod \"e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") " pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.804668 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.803676    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-envoy\") pod \"e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") " pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.827126 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.826253    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg5vk\" (UniqueName: \"kubernetes.io/projected/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-kube-api-access-wg5vk\") pod \"e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") " pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:58.827126 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:58.826556    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:22:59.129426 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:59.128752    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k"
Nov 03 20:22:59.129426 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:59.128899    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f"
Nov 03 20:22:59.220222 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:59.219733    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"]
Nov 03 20:22:59.222663 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:59.222581    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:22:59.222796 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:59.222687    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:22:59.222796 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:59.222721    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:22:59.293780 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:59.293733    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq" event={"ID":"436959f1-0deb-43cc-b8e0-ca75a6f0afbd","Type":"ContainerStarted","Data":"049931a7939314cd179d9e5e049c3a23400440eb1d4b9e8d561c537f95f9cd63"}
Nov 03 20:22:59.571627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:59.571405    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm"]
Nov 03 20:22:59.628232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:22:59.628191    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp"]
Nov 03 20:23:00.302631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:00.302562    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq" event={"ID":"436959f1-0deb-43cc-b8e0-ca75a6f0afbd","Type":"ContainerStarted","Data":"2a9a4ea1e1b460a1696e95d47270d6869143cb09f49d12564b190da7bd639219"}
Nov 03 20:23:00.329395 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:00.329344    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq" podStartSLOduration=2.329331959 podStartE2EDuration="2.329331959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:00.328621265 +0000 UTC m=+1310.461749728" watchObservedRunningTime="2026-11-03 20:23:00.329331959 +0000 UTC m=+1310.462460423"
Nov 03 20:23:00.836009 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:00.835976    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:23:00.857945 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:00.857915    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:23:01.310959 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:01.310931    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:23:01.312087 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:01.312071    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:23:03.738468 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:03.738419    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn"]
Nov 03 20:23:03.745086 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:03.745064    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn"
Nov 03 20:23:03.752877 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:03.752850    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-c4f356\""
Nov 03 20:23:03.759233 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:03.758503    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn"]
Nov 03 20:23:03.770912 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:03.766007    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j"]
Nov 03 20:23:03.771824 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:03.771779    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j"
Nov 03 20:23:03.788589 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:03.786582    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-c4f356-dockercfg-9r6jt\""
Nov 03 20:23:03.791045 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:03.791007    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j"]
Nov 03 20:23:03.802409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:03.802368    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22njh\" (UniqueName: \"kubernetes.io/projected/ae58dac5-e413-4ba5-8440-3d0b85bc9c7c-kube-api-access-22njh\") pod \"payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn\" (UID: \"ae58dac5-e413-4ba5-8440-3d0b85bc9c7c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn"
Nov 03 20:23:03.802576 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:03.802478    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35b29971-9827-4d3b-800a-4d05f84d6b5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j\" (UID: \"35b29971-9827-4d3b-800a-4d05f84d6b5d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j"
Nov 03 20:23:03.802576 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:03.802507    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae58dac5-e413-4ba5-8440-3d0b85bc9c7c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn\" (UID: \"ae58dac5-e413-4ba5-8440-3d0b85bc9c7c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn"
Nov 03 20:23:03.802923 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:03.802893    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phq24\" (UniqueName: \"kubernetes.io/projected/35b29971-9827-4d3b-800a-4d05f84d6b5d-kube-api-access-phq24\") pod \"payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j\" (UID: \"35b29971-9827-4d3b-800a-4d05f84d6b5d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j"
Nov 03 20:23:03.916662 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:03.908271    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phq24\" (UniqueName: \"kubernetes.io/projected/35b29971-9827-4d3b-800a-4d05f84d6b5d-kube-api-access-phq24\") pod \"payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j\" (UID: \"35b29971-9827-4d3b-800a-4d05f84d6b5d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j"
Nov 03 20:23:03.916662 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:03.908420    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22njh\" (UniqueName: \"kubernetes.io/projected/ae58dac5-e413-4ba5-8440-3d0b85bc9c7c-kube-api-access-22njh\") pod \"payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn\" (UID: \"ae58dac5-e413-4ba5-8440-3d0b85bc9c7c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn"
Nov 03 20:23:03.916662 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:03.908525    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35b29971-9827-4d3b-800a-4d05f84d6b5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j\" (UID: \"35b29971-9827-4d3b-800a-4d05f84d6b5d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j"
Nov 03 20:23:03.916662 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:03.908551    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae58dac5-e413-4ba5-8440-3d0b85bc9c7c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn\" (UID: \"ae58dac5-e413-4ba5-8440-3d0b85bc9c7c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn"
Nov 03 20:23:03.916662 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:03.909520    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae58dac5-e413-4ba5-8440-3d0b85bc9c7c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn\" (UID: \"ae58dac5-e413-4ba5-8440-3d0b85bc9c7c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn"
Nov 03 20:23:03.948646 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:03.921980    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35b29971-9827-4d3b-800a-4d05f84d6b5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j\" (UID: \"35b29971-9827-4d3b-800a-4d05f84d6b5d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j"
Nov 03 20:23:04.085716 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:04.085524    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22njh\" (UniqueName: \"kubernetes.io/projected/ae58dac5-e413-4ba5-8440-3d0b85bc9c7c-kube-api-access-22njh\") pod \"payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn\" (UID: \"ae58dac5-e413-4ba5-8440-3d0b85bc9c7c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn"
Nov 03 20:23:04.097943 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:04.092933    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phq24\" (UniqueName: \"kubernetes.io/projected/35b29971-9827-4d3b-800a-4d05f84d6b5d-kube-api-access-phq24\") pod \"payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j\" (UID: \"35b29971-9827-4d3b-800a-4d05f84d6b5d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j"
Nov 03 20:23:04.107943 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:04.107912    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j"
Nov 03 20:23:04.376357 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:04.376270    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn"
Nov 03 20:23:04.414909 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:04.414880    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j"]
Nov 03 20:23:04.415362 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:23:04.415337    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b29971_9827_4d3b_800a_4d05f84d6b5d.slice/crio-b006df236e084c9defcdd7b4a4fdcd90f5762975da2c548522e584a3bbed64a9 WatchSource:0}: Error finding container b006df236e084c9defcdd7b4a4fdcd90f5762975da2c548522e584a3bbed64a9: Status 404 returned error can't find the container with id b006df236e084c9defcdd7b4a4fdcd90f5762975da2c548522e584a3bbed64a9
Nov 03 20:23:04.572478 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:04.572444    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm" podUID="54bdf951-8d0e-4acc-bb6a-75cb74e2ae98" containerName="payload-processing" containerID="cri-o://e7dc80838428d7d8687d9117f9222295436f3595888b3f3d3f868a892f87ca6a" gracePeriod=25
Nov 03 20:23:04.629013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:04.628918    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp" podUID="24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c" containerName="payload-pre-processing" containerID="cri-o://909b400ced035fcd5228faabfc8d3b7f7507455fc89b03bb4f6c3415e6e09878" gracePeriod=25
Nov 03 20:23:04.681723 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:04.681690    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn"]
Nov 03 20:23:04.685399 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:23:04.685365    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae58dac5_e413_4ba5_8440_3d0b85bc9c7c.slice/crio-0bea94c899df4c030f968d494a0330f468891f73caea2e2534d77a19537fec39 WatchSource:0}: Error finding container 0bea94c899df4c030f968d494a0330f468891f73caea2e2534d77a19537fec39: Status 404 returned error can't find the container with id 0bea94c899df4c030f968d494a0330f468891f73caea2e2534d77a19537fec39
Nov 03 20:23:04.936170 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:04.935986    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp"
Nov 03 20:23:04.969801 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:04.969772    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm"
Nov 03 20:23:04.976396 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:04.975484    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c-plugins-config-volume\") pod \"24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c\" (UID: \"24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c\") "
Nov 03 20:23:04.976396 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:04.975653    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c-kube-api-access-vsr2v\" (UniqueName: \"kubernetes.io/projected/24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c-kube-api-access-vsr2v\") pod \"24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c\" (UID: \"24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c\") "
Nov 03 20:23:04.976396 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:04.976102    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c-plugins-config-volume" pod "24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c" (UID: "24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:04.978867 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:04.977684    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:04.981158 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:04.981135    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c-kube-api-access-vsr2v" pod "24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c" (UID: "24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c"). InnerVolumeSpecName "kube-api-access-vsr2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:05.079155 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.079121    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54bdf951-8d0e-4acc-bb6a-75cb74e2ae98-kube-api-access-zvvdk\" (UniqueName: \"kubernetes.io/projected/54bdf951-8d0e-4acc-bb6a-75cb74e2ae98-kube-api-access-zvvdk\") pod \"54bdf951-8d0e-4acc-bb6a-75cb74e2ae98\" (UID: \"54bdf951-8d0e-4acc-bb6a-75cb74e2ae98\") "
Nov 03 20:23:05.079334 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.079176    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54bdf951-8d0e-4acc-bb6a-75cb74e2ae98-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54bdf951-8d0e-4acc-bb6a-75cb74e2ae98-plugins-config-volume\") pod \"54bdf951-8d0e-4acc-bb6a-75cb74e2ae98\" (UID: \"54bdf951-8d0e-4acc-bb6a-75cb74e2ae98\") "
Nov 03 20:23:05.079679 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.079644    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54bdf951-8d0e-4acc-bb6a-75cb74e2ae98-plugins-config-volume" pod "54bdf951-8d0e-4acc-bb6a-75cb74e2ae98" (UID: "54bdf951-8d0e-4acc-bb6a-75cb74e2ae98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:05.080065 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.080047    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsr2v\" (UniqueName: \"kubernetes.io/projected/24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c-kube-api-access-vsr2v\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:05.080124 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.080069    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54bdf951-8d0e-4acc-bb6a-75cb74e2ae98-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:05.082070 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.082035    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54bdf951-8d0e-4acc-bb6a-75cb74e2ae98-kube-api-access-zvvdk" pod "54bdf951-8d0e-4acc-bb6a-75cb74e2ae98" (UID: "54bdf951-8d0e-4acc-bb6a-75cb74e2ae98"). InnerVolumeSpecName "kube-api-access-zvvdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:05.184167 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.184074    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvvdk\" (UniqueName: \"kubernetes.io/projected/54bdf951-8d0e-4acc-bb6a-75cb74e2ae98-kube-api-access-zvvdk\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:05.309035 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.309006    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8"
Nov 03 20:23:05.368862 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.368818    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn" event={"ID":"ae58dac5-e413-4ba5-8440-3d0b85bc9c7c","Type":"ContainerStarted","Data":"0b143b94844ab8fd1108906eee599673589871636cafd850c6d6609a7acb406b"}
Nov 03 20:23:05.369065 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.368884    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn" event={"ID":"ae58dac5-e413-4ba5-8440-3d0b85bc9c7c","Type":"ContainerStarted","Data":"0bea94c899df4c030f968d494a0330f468891f73caea2e2534d77a19537fec39"}
Nov 03 20:23:05.369065 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.368935    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn"
Nov 03 20:23:05.372077 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.371075    4421 generic.go:356] "Generic (PLEG): container finished" podID="54bdf951-8d0e-4acc-bb6a-75cb74e2ae98" containerID="e7dc80838428d7d8687d9117f9222295436f3595888b3f3d3f868a892f87ca6a" exitCode=0
Nov 03 20:23:05.372077 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.371142    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm"
Nov 03 20:23:05.372077 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.371186    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm" event={"ID":"54bdf951-8d0e-4acc-bb6a-75cb74e2ae98","Type":"ContainerDied","Data":"e7dc80838428d7d8687d9117f9222295436f3595888b3f3d3f868a892f87ca6a"}
Nov 03 20:23:05.372077 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.371228    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm" event={"ID":"54bdf951-8d0e-4acc-bb6a-75cb74e2ae98","Type":"ContainerDied","Data":"609e01486f6ba74101fa8d71913bd30356b3b96a1be55882838d84cf1faacd53"}
Nov 03 20:23:05.372077 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.371256    4421 scope.go:122] "RemoveContainer" containerID="e7dc80838428d7d8687d9117f9222295436f3595888b3f3d3f868a892f87ca6a"
Nov 03 20:23:05.375049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.375024    4421 generic.go:356] "Generic (PLEG): container finished" podID="24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c" containerID="909b400ced035fcd5228faabfc8d3b7f7507455fc89b03bb4f6c3415e6e09878" exitCode=0
Nov 03 20:23:05.375275 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.375254    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp" event={"ID":"24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c","Type":"ContainerDied","Data":"909b400ced035fcd5228faabfc8d3b7f7507455fc89b03bb4f6c3415e6e09878"}
Nov 03 20:23:05.375343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.375306    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp" event={"ID":"24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c","Type":"ContainerDied","Data":"b1f258370484585000a848559320643331c7beab57e7e7716b58b11bec6d0441"}
Nov 03 20:23:05.375403 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.375376    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp"
Nov 03 20:23:05.390590 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.390562    4421 scope.go:122] "RemoveContainer" containerID="e7dc80838428d7d8687d9117f9222295436f3595888b3f3d3f868a892f87ca6a"
Nov 03 20:23:05.391850 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:23:05.391480    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7dc80838428d7d8687d9117f9222295436f3595888b3f3d3f868a892f87ca6a\": container with ID starting with e7dc80838428d7d8687d9117f9222295436f3595888b3f3d3f868a892f87ca6a not found: ID does not exist" containerID="e7dc80838428d7d8687d9117f9222295436f3595888b3f3d3f868a892f87ca6a"
Nov 03 20:23:05.391850 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.391514    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7dc80838428d7d8687d9117f9222295436f3595888b3f3d3f868a892f87ca6a"} err="failed to get container status \"e7dc80838428d7d8687d9117f9222295436f3595888b3f3d3f868a892f87ca6a\": rpc error: code = NotFound desc = could not find container \"e7dc80838428d7d8687d9117f9222295436f3595888b3f3d3f868a892f87ca6a\": container with ID starting with e7dc80838428d7d8687d9117f9222295436f3595888b3f3d3f868a892f87ca6a not found: ID does not exist"
Nov 03 20:23:05.391850 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.391539    4421 scope.go:122] "RemoveContainer" containerID="909b400ced035fcd5228faabfc8d3b7f7507455fc89b03bb4f6c3415e6e09878"
Nov 03 20:23:05.393772 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.393026    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j" event={"ID":"35b29971-9827-4d3b-800a-4d05f84d6b5d","Type":"ContainerStarted","Data":"b301df9685ef04a70fff1d0cfa391de06fd7bd656088233c193cedd0ff50bc11"}
Nov 03 20:23:05.393772 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.393082    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j" event={"ID":"35b29971-9827-4d3b-800a-4d05f84d6b5d","Type":"ContainerStarted","Data":"b006df236e084c9defcdd7b4a4fdcd90f5762975da2c548522e584a3bbed64a9"}
Nov 03 20:23:05.393772 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.393708    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j"
Nov 03 20:23:05.405893 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.398383    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn" podStartSLOduration=2.398266069 podStartE2EDuration="2.398266069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:05.3908723 +0000 UTC m=+1315.524000762" watchObservedRunningTime="2026-11-03 20:23:05.398266069 +0000 UTC m=+1315.531394532"
Nov 03 20:23:05.429906 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.429878    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm"]
Nov 03 20:23:05.431585 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.431558    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1142fd-7fd577747d-x9fvm"]
Nov 03 20:23:05.432343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.432329    4421 scope.go:122] "RemoveContainer" containerID="909b400ced035fcd5228faabfc8d3b7f7507455fc89b03bb4f6c3415e6e09878"
Nov 03 20:23:05.432881 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:23:05.432853    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"909b400ced035fcd5228faabfc8d3b7f7507455fc89b03bb4f6c3415e6e09878\": container with ID starting with 909b400ced035fcd5228faabfc8d3b7f7507455fc89b03bb4f6c3415e6e09878 not found: ID does not exist" containerID="909b400ced035fcd5228faabfc8d3b7f7507455fc89b03bb4f6c3415e6e09878"
Nov 03 20:23:05.432955 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.432892    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"909b400ced035fcd5228faabfc8d3b7f7507455fc89b03bb4f6c3415e6e09878"} err="failed to get container status \"909b400ced035fcd5228faabfc8d3b7f7507455fc89b03bb4f6c3415e6e09878\": rpc error: code = NotFound desc = could not find container \"909b400ced035fcd5228faabfc8d3b7f7507455fc89b03bb4f6c3415e6e09878\": container with ID starting with 909b400ced035fcd5228faabfc8d3b7f7507455fc89b03bb4f6c3415e6e09878 not found: ID does not exist"
Nov 03 20:23:05.442546 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.442520    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp"]
Nov 03 20:23:05.446670 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.446647    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-54964f48c-6xlsp"]
Nov 03 20:23:05.467353 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:05.467309    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j" podStartSLOduration=2.467297207 podStartE2EDuration="2.467297207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:05.464535807 +0000 UTC m=+1315.597664290" watchObservedRunningTime="2026-11-03 20:23:05.467297207 +0000 UTC m=+1315.600425668"
Nov 03 20:23:06.067194 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:06.067114    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c" path="/var/lib/kubelet/pods/24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c/volumes"
Nov 03 20:23:06.068721 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:06.068695    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54bdf951-8d0e-4acc-bb6a-75cb74e2ae98" path="/var/lib/kubelet/pods/54bdf951-8d0e-4acc-bb6a-75cb74e2ae98/volumes"
Nov 03 20:23:06.494316 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:06.494275    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb"]
Nov 03 20:23:06.495168 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:06.495127    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb" podUID="1007c757-280e-4436-8c9f-e5ac5deda164" containerName="maas-api" containerID="cri-o://9414ca6a6aa63a7151776550aa5de690d4e14c009c5d23efe5f3fd2eb5762b45" gracePeriod=30
Nov 03 20:23:07.011037 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.011010    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb"
Nov 03 20:23:07.072455 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.072420    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1007c757-280e-4436-8c9f-e5ac5deda164-kube-api-access-2lrtj\" (UniqueName: \"kubernetes.io/projected/1007c757-280e-4436-8c9f-e5ac5deda164-kube-api-access-2lrtj\") pod \"1007c757-280e-4436-8c9f-e5ac5deda164\" (UID: \"1007c757-280e-4436-8c9f-e5ac5deda164\") "
Nov 03 20:23:07.073009 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.072565    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1007c757-280e-4436-8c9f-e5ac5deda164-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1007c757-280e-4436-8c9f-e5ac5deda164-openshift-service-ca\") pod \"1007c757-280e-4436-8c9f-e5ac5deda164\" (UID: \"1007c757-280e-4436-8c9f-e5ac5deda164\") "
Nov 03 20:23:07.073009 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.072831    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1007c757-280e-4436-8c9f-e5ac5deda164-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1007c757-280e-4436-8c9f-e5ac5deda164-maas-api-tls\") pod \"1007c757-280e-4436-8c9f-e5ac5deda164\" (UID: \"1007c757-280e-4436-8c9f-e5ac5deda164\") "
Nov 03 20:23:07.073009 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.072996    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1007c757-280e-4436-8c9f-e5ac5deda164-openshift-service-ca" pod "1007c757-280e-4436-8c9f-e5ac5deda164" (UID: "1007c757-280e-4436-8c9f-e5ac5deda164"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:07.073238 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.073216    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1007c757-280e-4436-8c9f-e5ac5deda164-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1007c757-280e-4436-8c9f-e5ac5deda164-metrics-certs\") pod \"1007c757-280e-4436-8c9f-e5ac5deda164\" (UID: \"1007c757-280e-4436-8c9f-e5ac5deda164\") "
Nov 03 20:23:07.074447 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.074271    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1007c757-280e-4436-8c9f-e5ac5deda164-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:07.076134 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.076105    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1007c757-280e-4436-8c9f-e5ac5deda164-maas-api-tls" pod "1007c757-280e-4436-8c9f-e5ac5deda164" (UID: "1007c757-280e-4436-8c9f-e5ac5deda164"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:23:07.077845 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.077819    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1007c757-280e-4436-8c9f-e5ac5deda164-metrics-certs" pod "1007c757-280e-4436-8c9f-e5ac5deda164" (UID: "1007c757-280e-4436-8c9f-e5ac5deda164"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:23:07.078700 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.078674    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1007c757-280e-4436-8c9f-e5ac5deda164-kube-api-access-2lrtj" pod "1007c757-280e-4436-8c9f-e5ac5deda164" (UID: "1007c757-280e-4436-8c9f-e5ac5deda164"). InnerVolumeSpecName "kube-api-access-2lrtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:07.216970 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.215218    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1007c757-280e-4436-8c9f-e5ac5deda164-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:07.216970 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.215257    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1007c757-280e-4436-8c9f-e5ac5deda164-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:07.216970 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.215272    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lrtj\" (UniqueName: \"kubernetes.io/projected/1007c757-280e-4436-8c9f-e5ac5deda164-kube-api-access-2lrtj\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:07.419367 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.419330    4421 generic.go:356] "Generic (PLEG): container finished" podID="1007c757-280e-4436-8c9f-e5ac5deda164" containerID="9414ca6a6aa63a7151776550aa5de690d4e14c009c5d23efe5f3fd2eb5762b45" exitCode=0
Nov 03 20:23:07.419701 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.419453    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb"
Nov 03 20:23:07.419701 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.419471    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb" event={"ID":"1007c757-280e-4436-8c9f-e5ac5deda164","Type":"ContainerDied","Data":"9414ca6a6aa63a7151776550aa5de690d4e14c009c5d23efe5f3fd2eb5762b45"}
Nov 03 20:23:07.419701 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.419537    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb" event={"ID":"1007c757-280e-4436-8c9f-e5ac5deda164","Type":"ContainerDied","Data":"4a3771ae8fd95257bd8a30e3d634ee6103333e8d4b865b78eeb3ccdc6660939f"}
Nov 03 20:23:07.419701 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.419586    4421 scope.go:122] "RemoveContainer" containerID="9414ca6a6aa63a7151776550aa5de690d4e14c009c5d23efe5f3fd2eb5762b45"
Nov 03 20:23:07.442253 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.442229    4421 scope.go:122] "RemoveContainer" containerID="9414ca6a6aa63a7151776550aa5de690d4e14c009c5d23efe5f3fd2eb5762b45"
Nov 03 20:23:07.442651 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:23:07.442580    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9414ca6a6aa63a7151776550aa5de690d4e14c009c5d23efe5f3fd2eb5762b45\": container with ID starting with 9414ca6a6aa63a7151776550aa5de690d4e14c009c5d23efe5f3fd2eb5762b45 not found: ID does not exist" containerID="9414ca6a6aa63a7151776550aa5de690d4e14c009c5d23efe5f3fd2eb5762b45"
Nov 03 20:23:07.442746 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.442664    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9414ca6a6aa63a7151776550aa5de690d4e14c009c5d23efe5f3fd2eb5762b45"} err="failed to get container status \"9414ca6a6aa63a7151776550aa5de690d4e14c009c5d23efe5f3fd2eb5762b45\": rpc error: code = NotFound desc = could not find container \"9414ca6a6aa63a7151776550aa5de690d4e14c009c5d23efe5f3fd2eb5762b45\": container with ID starting with 9414ca6a6aa63a7151776550aa5de690d4e14c009c5d23efe5f3fd2eb5762b45 not found: ID does not exist"
Nov 03 20:23:07.461529 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.461500    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb"]
Nov 03 20:23:07.471781 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.471230    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf9b0a-77b6977bd-7b7kb"]
Nov 03 20:23:07.681684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.681650    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"]
Nov 03 20:23:07.689301 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.689270    4421 cpu_manager.go:414] "containerMap: removing container" podUID="1007c757-280e-4436-8c9f-e5ac5deda164" containerName="maas-api"
Nov 03 20:23:07.689502 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.689486    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1007c757-280e-4436-8c9f-e5ac5deda164" containerName="maas-api"
Nov 03 20:23:07.689712 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.689697    4421 cpu_manager.go:414] "containerMap: removing container" podUID="54bdf951-8d0e-4acc-bb6a-75cb74e2ae98" containerName="payload-processing"
Nov 03 20:23:07.689820 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.689806    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54bdf951-8d0e-4acc-bb6a-75cb74e2ae98" containerName="payload-processing"
Nov 03 20:23:07.689944 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.689932    4421 cpu_manager.go:414] "containerMap: removing container" podUID="24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c" containerName="payload-pre-processing"
Nov 03 20:23:07.690029 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.690019    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c" containerName="payload-pre-processing"
Nov 03 20:23:07.690315 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.690299    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="24ada5c2-1dd7-442b-9ab2-b8e6e3b2f97c" containerName="payload-pre-processing"
Nov 03 20:23:07.690418 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.690409    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="1007c757-280e-4436-8c9f-e5ac5deda164" containerName="maas-api"
Nov 03 20:23:07.690493 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.690485    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="54bdf951-8d0e-4acc-bb6a-75cb74e2ae98" containerName="payload-processing"
Nov 03 20:23:07.699636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.699582    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"]
Nov 03 20:23:07.699962 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.699949    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:07.708737 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.708653    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-1b646f-openshift-default-dockercfg-df44n\""
Nov 03 20:23:07.846956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.835554    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-token\") pod \"e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") " pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:07.846956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.835660    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-envoy\") pod \"e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") " pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:07.846956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.835743    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-credential-socket\") pod \"e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") " pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:07.846956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.835797    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7gzc\" (UniqueName: \"kubernetes.io/projected/aab9be1c-04f0-4285-9e31-fbb87ec0da02-kube-api-access-w7gzc\") pod \"e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") " pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:07.846956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.835872    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-workload-certs\") pod \"e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") " pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:07.846956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.835955    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-podinfo\") pod \"e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") " pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:07.846956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.836046    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istiod-ca-cert\") pod \"e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") " pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:07.846956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.836092    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-data\") pod \"e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") " pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:07.846956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.836144    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-workload-socket\") pod \"e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") " pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:07.937836 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.937806    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-token\") pod \"e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") " pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:07.939991 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.937873    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-envoy\") pod \"e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") " pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:07.939991 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.937923    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-credential-socket\") pod \"e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") " pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:07.939991 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.937965    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7gzc\" (UniqueName: \"kubernetes.io/projected/aab9be1c-04f0-4285-9e31-fbb87ec0da02-kube-api-access-w7gzc\") pod \"e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") " pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:07.939991 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.938010    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-workload-certs\") pod \"e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") " pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:07.939991 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.938390    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-podinfo\") pod \"e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") " pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:07.939991 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.938452    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-workload-certs\") pod \"e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") " pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:07.939991 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.938460    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istiod-ca-cert\") pod \"e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") " pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:07.939991 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.938395    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-credential-socket\") pod \"e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") " pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:07.939991 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.938515    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-data\") pod \"e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") " pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:07.939991 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.938553    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-workload-socket\") pod \"e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") " pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:07.939991 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.938919    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-data\") pod \"e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") " pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:07.939991 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.938927    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-workload-socket\") pod \"e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") " pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:07.939991 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.939061    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istiod-ca-cert\") pod \"e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") " pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:07.941155 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.941135    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-envoy\") pod \"e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") " pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:07.941250 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.941173    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-podinfo\") pod \"e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") " pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:07.957199 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.957177    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-token\") pod \"e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") " pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:07.957292 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:07.957221    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7gzc\" (UniqueName: \"kubernetes.io/projected/aab9be1c-04f0-4285-9e31-fbb87ec0da02-kube-api-access-w7gzc\") pod \"e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") " pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:08.053526 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:08.053444    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1007c757-280e-4436-8c9f-e5ac5deda164" path="/var/lib/kubelet/pods/1007c757-280e-4436-8c9f-e5ac5deda164/volumes"
Nov 03 20:23:08.095196 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:08.095167    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:08.460254 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:08.460217    4421 generic.go:356] "Generic (PLEG): container finished" podID="c263b0e4-5dd7-4a46-a527-f461b779e42f" containerID="01a6879065dc2564a3816015282a6cd62df4f2f7a9817ab678edc8184ca4a60e" exitCode=0
Nov 03 20:23:08.460581 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:08.460556    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn" event={"ID":"c263b0e4-5dd7-4a46-a527-f461b779e42f","Type":"ContainerDied","Data":"01a6879065dc2564a3816015282a6cd62df4f2f7a9817ab678edc8184ca4a60e"}
Nov 03 20:23:08.461280 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:08.461257    4421 scope.go:122] "RemoveContainer" containerID="01a6879065dc2564a3816015282a6cd62df4f2f7a9817ab678edc8184ca4a60e"
Nov 03 20:23:08.464191 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:08.464152    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:23:08.464401 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:08.464381    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:23:08.464552 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:08.464532    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:23:08.551844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:08.551814    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"]
Nov 03 20:23:08.559915 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:08.559866    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:23:08.560035 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:08.559965    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:23:08.560035 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:08.560011    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:23:09.491140 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:09.491109    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn" event={"ID":"c263b0e4-5dd7-4a46-a527-f461b779e42f","Type":"ContainerStarted","Data":"209fa2c38f7bf6ea798eb4967effe419aa32361835bfaffbb944c41e72897d87"}
Nov 03 20:23:09.492787 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:09.492764    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz" event={"ID":"aab9be1c-04f0-4285-9e31-fbb87ec0da02","Type":"ContainerStarted","Data":"9e1a289410f2541b26b444264e777710cda813c5c8a70e6829a3b6aa0caced76"}
Nov 03 20:23:09.492921 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:09.492803    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz" event={"ID":"aab9be1c-04f0-4285-9e31-fbb87ec0da02","Type":"ContainerStarted","Data":"187eb0cc0e4e9b18b2170ec46365099b69cd6872736da9a5dc98a4a64dbef82c"}
Nov 03 20:23:09.540191 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:09.540150    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz" podStartSLOduration=2.540136813 podStartE2EDuration="2.540136813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:09.538567134 +0000 UTC m=+1319.671695624" watchObservedRunningTime="2026-11-03 20:23:09.540136813 +0000 UTC m=+1319.673265275"
Nov 03 20:23:09.865745 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:09.865701    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:23:09.866512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:09.866490    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 03 20:23:09.866653 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:09.866544    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn" podUID="c263b0e4-5dd7-4a46-a527-f461b779e42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 03 20:23:09.890247 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:09.890217    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28"]
Nov 03 20:23:09.894999 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:09.894982    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28"
Nov 03 20:23:09.900147 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:09.900120    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1142fd-dockercfg-zzpk6\""
Nov 03 20:23:09.900264 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:09.900213    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1142fd\""
Nov 03 20:23:09.905292 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:09.905269    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28"]
Nov 03 20:23:10.031633 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:10.005527    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d787a81-bb04-461a-861d-e81e1688d5a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28\" (UID: \"0d787a81-bb04-461a-861d-e81e1688d5a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28"
Nov 03 20:23:10.032223 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:10.032185    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r99h5\" (UniqueName: \"kubernetes.io/projected/0d787a81-bb04-461a-861d-e81e1688d5a7-kube-api-access-r99h5\") pod \"payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28\" (UID: \"0d787a81-bb04-461a-861d-e81e1688d5a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28"
Nov 03 20:23:10.251721 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:10.247900    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r99h5\" (UniqueName: \"kubernetes.io/projected/0d787a81-bb04-461a-861d-e81e1688d5a7-kube-api-access-r99h5\") pod \"payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28\" (UID: \"0d787a81-bb04-461a-861d-e81e1688d5a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28"
Nov 03 20:23:10.251721 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:10.248088    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d787a81-bb04-461a-861d-e81e1688d5a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28\" (UID: \"0d787a81-bb04-461a-861d-e81e1688d5a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28"
Nov 03 20:23:10.266305 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:10.265883    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1142fd\""
Nov 03 20:23:10.288917 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:10.271475    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d787a81-bb04-461a-861d-e81e1688d5a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28\" (UID: \"0d787a81-bb04-461a-861d-e81e1688d5a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28"
Nov 03 20:23:10.302160 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:10.298835    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:10.302160 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:10.298885    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl"]
Nov 03 20:23:10.303387 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:10.303303    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r99h5\" (UniqueName: \"kubernetes.io/projected/0d787a81-bb04-461a-861d-e81e1688d5a7-kube-api-access-r99h5\") pod \"payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28\" (UID: \"0d787a81-bb04-461a-861d-e81e1688d5a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28"
Nov 03 20:23:10.307571 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:10.307557    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl"]
Nov 03 20:23:10.307710 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:10.307695    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:23:10.307890 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:10.307873    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:10.308086 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:10.307780    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl"
Nov 03 20:23:10.469751 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:10.469713    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3fd1d16-f04c-4807-8991-1987e5245fb8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl\" (UID: \"d3fd1d16-f04c-4807-8991-1987e5245fb8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl"
Nov 03 20:23:10.470059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:10.470023    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57wsq\" (UniqueName: \"kubernetes.io/projected/d3fd1d16-f04c-4807-8991-1987e5245fb8-kube-api-access-57wsq\") pod \"payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl\" (UID: \"d3fd1d16-f04c-4807-8991-1987e5245fb8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl"
Nov 03 20:23:10.500840 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:10.500802    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:10.501860 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:10.501801    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:23:10.515740 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:10.515722    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1142fd-dockercfg-zzpk6\""
Nov 03 20:23:10.523826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:10.523811    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28"
Nov 03 20:23:10.592115 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:10.592079    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3fd1d16-f04c-4807-8991-1987e5245fb8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl\" (UID: \"d3fd1d16-f04c-4807-8991-1987e5245fb8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl"
Nov 03 20:23:10.592299 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:10.592200    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57wsq\" (UniqueName: \"kubernetes.io/projected/d3fd1d16-f04c-4807-8991-1987e5245fb8-kube-api-access-57wsq\") pod \"payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl\" (UID: \"d3fd1d16-f04c-4807-8991-1987e5245fb8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl"
Nov 03 20:23:10.593224 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:10.593189    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3fd1d16-f04c-4807-8991-1987e5245fb8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl\" (UID: \"d3fd1d16-f04c-4807-8991-1987e5245fb8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl"
Nov 03 20:23:10.605066 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:10.605043    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57wsq\" (UniqueName: \"kubernetes.io/projected/d3fd1d16-f04c-4807-8991-1987e5245fb8-kube-api-access-57wsq\") pod \"payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl\" (UID: \"d3fd1d16-f04c-4807-8991-1987e5245fb8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl"
Nov 03 20:23:10.714146 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:10.697451    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl"
Nov 03 20:23:10.872862 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:10.872776    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 03 20:23:10.872862 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:10.872835    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn" podUID="c263b0e4-5dd7-4a46-a527-f461b779e42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 03 20:23:11.007666 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:11.007642    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28"]
Nov 03 20:23:11.010244 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:23:11.010194    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d787a81_bb04_461a_861d_e81e1688d5a7.slice/crio-a4f12321a3ffc3025508a996ae0b0a27652de5f7ddf8ec087eda6dbed33492a6 WatchSource:0}: Error finding container a4f12321a3ffc3025508a996ae0b0a27652de5f7ddf8ec087eda6dbed33492a6: Status 404 returned error can't find the container with id a4f12321a3ffc3025508a996ae0b0a27652de5f7ddf8ec087eda6dbed33492a6
Nov 03 20:23:11.111768 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:23:11.110954    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3fd1d16_f04c_4807_8991_1987e5245fb8.slice/crio-13c650ba7931dbbda094c77d65b3cb475d6a4e5ed502a1d49a4b1472c3d7cf42 WatchSource:0}: Error finding container 13c650ba7931dbbda094c77d65b3cb475d6a4e5ed502a1d49a4b1472c3d7cf42: Status 404 returned error can't find the container with id 13c650ba7931dbbda094c77d65b3cb475d6a4e5ed502a1d49a4b1472c3d7cf42
Nov 03 20:23:11.112114 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:11.112091    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl"]
Nov 03 20:23:11.651627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:11.651316    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl" event={"ID":"d3fd1d16-f04c-4807-8991-1987e5245fb8","Type":"ContainerStarted","Data":"e6f765617667051d74514a60bdd41c7f974bb76af2acba589af39f827e77d94d"}
Nov 03 20:23:11.651627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:11.651384    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl" event={"ID":"d3fd1d16-f04c-4807-8991-1987e5245fb8","Type":"ContainerStarted","Data":"13c650ba7931dbbda094c77d65b3cb475d6a4e5ed502a1d49a4b1472c3d7cf42"}
Nov 03 20:23:11.655617 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:11.652573    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl"
Nov 03 20:23:11.671618 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:11.671546    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl" podStartSLOduration=2.6715292379999998 podStartE2EDuration="2.671529238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:11.671327826 +0000 UTC m=+1321.804456289" watchObservedRunningTime="2026-11-03 20:23:11.671529238 +0000 UTC m=+1321.804657701"
Nov 03 20:23:11.694624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:11.691257    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28" event={"ID":"0d787a81-bb04-461a-861d-e81e1688d5a7","Type":"ContainerStarted","Data":"4f37a359adb5ae0177ad0fb2ed1f8cd2b58d73fe09ba4de538ee0337b828607f"}
Nov 03 20:23:11.694624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:11.691334    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28" event={"ID":"0d787a81-bb04-461a-861d-e81e1688d5a7","Type":"ContainerStarted","Data":"a4f12321a3ffc3025508a996ae0b0a27652de5f7ddf8ec087eda6dbed33492a6"}
Nov 03 20:23:11.694624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:11.691385    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28"
Nov 03 20:23:11.715135 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:11.715006    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28" podStartSLOduration=2.714987996 podStartE2EDuration="2.714987996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:11.712896846 +0000 UTC m=+1321.846025309" watchObservedRunningTime="2026-11-03 20:23:11.714987996 +0000 UTC m=+1321.848116463"
Nov 03 20:23:11.868653 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:11.864202    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 03 20:23:11.868653 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:11.864266    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn" podUID="c263b0e4-5dd7-4a46-a527-f461b779e42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 03 20:23:12.866632 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:12.866007    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 03 20:23:12.866632 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:12.866074    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn" podUID="c263b0e4-5dd7-4a46-a527-f461b779e42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 03 20:23:13.381411 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:13.381375    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj"]
Nov 03 20:23:13.388261 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:13.388235    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj"
Nov 03 20:23:13.393050 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:13.393007    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj"]
Nov 03 20:23:13.394507 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:13.394464    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-1b646f-dockercfg-bwths\""
Nov 03 20:23:13.394708 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:13.394485    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj\" (UID: \"a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj"
Nov 03 20:23:13.394950 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:13.394880    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlf6j\" (UniqueName: \"kubernetes.io/projected/a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5-kube-api-access-xlf6j\") pod \"payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj\" (UID: \"a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj"
Nov 03 20:23:13.452746 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:13.447937    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-1b646f\""
Nov 03 20:23:13.531624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:13.528968    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj\" (UID: \"a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj"
Nov 03 20:23:13.531624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:13.529292    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlf6j\" (UniqueName: \"kubernetes.io/projected/a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5-kube-api-access-xlf6j\") pod \"payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj\" (UID: \"a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj"
Nov 03 20:23:13.531624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:13.530726    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj\" (UID: \"a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj"
Nov 03 20:23:13.627622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:13.605219    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlf6j\" (UniqueName: \"kubernetes.io/projected/a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5-kube-api-access-xlf6j\") pod \"payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj\" (UID: \"a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj"
Nov 03 20:23:13.737009 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:13.730187    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj"
Nov 03 20:23:13.868630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:13.865051    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 03 20:23:13.868630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:13.865113    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn" podUID="c263b0e4-5dd7-4a46-a527-f461b779e42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 03 20:23:14.238712 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:23:14.238674    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4eb3e18_16aa_4c22_92c2_c3dd2c1ac3d5.slice/crio-3b31d8c683102130cea0f06ca90a816fe79b7aba3caef32e4a555406e8c318e0 WatchSource:0}: Error finding container 3b31d8c683102130cea0f06ca90a816fe79b7aba3caef32e4a555406e8c318e0: Status 404 returned error can't find the container with id 3b31d8c683102130cea0f06ca90a816fe79b7aba3caef32e4a555406e8c318e0
Nov 03 20:23:14.239128 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:14.239104    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj"]
Nov 03 20:23:14.866626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:14.863014    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 03 20:23:14.866626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:14.863074    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn" podUID="c263b0e4-5dd7-4a46-a527-f461b779e42f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 03 20:23:14.884182 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:14.884142    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj" event={"ID":"a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5","Type":"ContainerStarted","Data":"6d6ab5611ab9b5e9766edbe1bc297fea9b7374ed8823260c6a97273b5ddce19f"}
Nov 03 20:23:14.884730 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:14.884208    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj"
Nov 03 20:23:14.884730 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:14.884232    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj" event={"ID":"a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5","Type":"ContainerStarted","Data":"3b31d8c683102130cea0f06ca90a816fe79b7aba3caef32e4a555406e8c318e0"}
Nov 03 20:23:14.906019 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:14.905968    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj" podStartSLOduration=1.9059542889999999 podStartE2EDuration="1.905954289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:14.902744567 +0000 UTC m=+1325.035873008" watchObservedRunningTime="2026-11-03 20:23:14.905954289 +0000 UTC m=+1325.039082779"
Nov 03 20:23:16.052264 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:16.051962    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:23:16.054088 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:16.054065    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:23:16.341533 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:16.341457    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv"]
Nov 03 20:23:16.350695 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:16.347580    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv"
Nov 03 20:23:16.352172 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:16.352142    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-c4f356-serving-cert\""
Nov 03 20:23:16.357925 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:16.357854    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv"]
Nov 03 20:23:16.407388 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:16.407360    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn"
Nov 03 20:23:16.464644 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:16.463418    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xxw6\" (UniqueName: \"kubernetes.io/projected/f479675b-17b7-432b-aef4-9cbc3bfb7359-kube-api-access-4xxw6\") pod \"maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv\" (UID: \"f479675b-17b7-432b-aef4-9cbc3bfb7359\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv"
Nov 03 20:23:16.464644 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:16.463527    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f479675b-17b7-432b-aef4-9cbc3bfb7359-metrics-certs\") pod \"maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv\" (UID: \"f479675b-17b7-432b-aef4-9cbc3bfb7359\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv"
Nov 03 20:23:16.464644 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:16.463661    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f479675b-17b7-432b-aef4-9cbc3bfb7359-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv\" (UID: \"f479675b-17b7-432b-aef4-9cbc3bfb7359\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv"
Nov 03 20:23:16.464644 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:16.463695    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f479675b-17b7-432b-aef4-9cbc3bfb7359-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv\" (UID: \"f479675b-17b7-432b-aef4-9cbc3bfb7359\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv"
Nov 03 20:23:16.565484 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:16.565445    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xxw6\" (UniqueName: \"kubernetes.io/projected/f479675b-17b7-432b-aef4-9cbc3bfb7359-kube-api-access-4xxw6\") pod \"maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv\" (UID: \"f479675b-17b7-432b-aef4-9cbc3bfb7359\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv"
Nov 03 20:23:16.565690 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:16.565555    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f479675b-17b7-432b-aef4-9cbc3bfb7359-metrics-certs\") pod \"maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv\" (UID: \"f479675b-17b7-432b-aef4-9cbc3bfb7359\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv"
Nov 03 20:23:16.565794 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:16.565696    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f479675b-17b7-432b-aef4-9cbc3bfb7359-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv\" (UID: \"f479675b-17b7-432b-aef4-9cbc3bfb7359\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv"
Nov 03 20:23:16.565794 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:16.565738    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f479675b-17b7-432b-aef4-9cbc3bfb7359-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv\" (UID: \"f479675b-17b7-432b-aef4-9cbc3bfb7359\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv"
Nov 03 20:23:16.566417 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:16.566387    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f479675b-17b7-432b-aef4-9cbc3bfb7359-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv\" (UID: \"f479675b-17b7-432b-aef4-9cbc3bfb7359\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv"
Nov 03 20:23:16.568529 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:16.568501    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f479675b-17b7-432b-aef4-9cbc3bfb7359-metrics-certs\") pod \"maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv\" (UID: \"f479675b-17b7-432b-aef4-9cbc3bfb7359\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv"
Nov 03 20:23:16.568783 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:16.568762    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f479675b-17b7-432b-aef4-9cbc3bfb7359-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv\" (UID: \"f479675b-17b7-432b-aef4-9cbc3bfb7359\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv"
Nov 03 20:23:16.578689 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:16.578667    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xxw6\" (UniqueName: \"kubernetes.io/projected/f479675b-17b7-432b-aef4-9cbc3bfb7359-kube-api-access-4xxw6\") pod \"maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv\" (UID: \"f479675b-17b7-432b-aef4-9cbc3bfb7359\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv"
Nov 03 20:23:16.682732 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:16.681000    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv"
Nov 03 20:23:17.052738 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:17.052711    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv"]
Nov 03 20:23:17.053238 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:23:17.053214    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf479675b_17b7_432b_aef4_9cbc3bfb7359.slice/crio-a194613d02dea7a4372d3a1eb13a45c1ca418091e602eda0c0f57291f604bdf4 WatchSource:0}: Error finding container a194613d02dea7a4372d3a1eb13a45c1ca418091e602eda0c0f57291f604bdf4: Status 404 returned error can't find the container with id a194613d02dea7a4372d3a1eb13a45c1ca418091e602eda0c0f57291f604bdf4
Nov 03 20:23:17.424104 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:17.424034    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j"
Nov 03 20:23:18.190318 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:18.190233    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv" podStartSLOduration=1.784337404 podStartE2EDuration="2.190212019s" totalImagesPullingTime="405.874615ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:18.177384925 +0000 UTC m=+1328.310513424" watchObservedRunningTime="2026-11-03 20:23:18.190212019 +0000 UTC m=+1328.323340482"
Nov 03 20:23:18.240823 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:18.232483    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv" event={"ID":"f479675b-17b7-432b-aef4-9cbc3bfb7359","Type":"ContainerStarted","Data":"23b5c1007d4b26ed38f71376dc5e8e21b66eaca3256c8a1b326ae3ba4431b74c"}
Nov 03 20:23:18.241681 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:18.241640    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv" event={"ID":"f479675b-17b7-432b-aef4-9cbc3bfb7359","Type":"ContainerStarted","Data":"a194613d02dea7a4372d3a1eb13a45c1ca418091e602eda0c0f57291f604bdf4"}
Nov 03 20:23:18.241936 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:18.241919    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv"
Nov 03 20:23:19.687762 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:23:19.687726    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:23:22.698981 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:22.698950    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28"
Nov 03 20:23:23.876248 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:23.876219    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl"
Nov 03 20:23:25.146979 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:25.146952    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv"
Nov 03 20:23:25.930677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:25.930639    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p"]
Nov 03 20:23:25.937798 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:25.937773    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj"
Nov 03 20:23:25.937964 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:25.937847    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p"
Nov 03 20:23:25.944128 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:25.944101    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-1b646f-serving-cert\""
Nov 03 20:23:25.956650 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:25.955853    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p"]
Nov 03 20:23:26.131622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:26.129429    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dba43a0-1306-49e2-8f2c-bc5e221381d2-metrics-certs\") pod \"maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p\" (UID: \"4dba43a0-1306-49e2-8f2c-bc5e221381d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p"
Nov 03 20:23:26.131622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:26.129490    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwjfc\" (UniqueName: \"kubernetes.io/projected/4dba43a0-1306-49e2-8f2c-bc5e221381d2-kube-api-access-dwjfc\") pod \"maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p\" (UID: \"4dba43a0-1306-49e2-8f2c-bc5e221381d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p"
Nov 03 20:23:26.131622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:26.129854    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dba43a0-1306-49e2-8f2c-bc5e221381d2-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p\" (UID: \"4dba43a0-1306-49e2-8f2c-bc5e221381d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p"
Nov 03 20:23:26.131622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:26.129894    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dba43a0-1306-49e2-8f2c-bc5e221381d2-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p\" (UID: \"4dba43a0-1306-49e2-8f2c-bc5e221381d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p"
Nov 03 20:23:26.232181 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:26.232129    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dba43a0-1306-49e2-8f2c-bc5e221381d2-metrics-certs\") pod \"maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p\" (UID: \"4dba43a0-1306-49e2-8f2c-bc5e221381d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p"
Nov 03 20:23:26.232181 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:26.232174    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwjfc\" (UniqueName: \"kubernetes.io/projected/4dba43a0-1306-49e2-8f2c-bc5e221381d2-kube-api-access-dwjfc\") pod \"maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p\" (UID: \"4dba43a0-1306-49e2-8f2c-bc5e221381d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p"
Nov 03 20:23:26.232639 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:26.232354    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dba43a0-1306-49e2-8f2c-bc5e221381d2-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p\" (UID: \"4dba43a0-1306-49e2-8f2c-bc5e221381d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p"
Nov 03 20:23:26.232639 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:26.232384    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dba43a0-1306-49e2-8f2c-bc5e221381d2-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p\" (UID: \"4dba43a0-1306-49e2-8f2c-bc5e221381d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p"
Nov 03 20:23:26.233103 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:26.233020    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dba43a0-1306-49e2-8f2c-bc5e221381d2-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p\" (UID: \"4dba43a0-1306-49e2-8f2c-bc5e221381d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p"
Nov 03 20:23:26.234950 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:26.234931    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dba43a0-1306-49e2-8f2c-bc5e221381d2-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p\" (UID: \"4dba43a0-1306-49e2-8f2c-bc5e221381d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p"
Nov 03 20:23:26.235050 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:26.234971    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dba43a0-1306-49e2-8f2c-bc5e221381d2-metrics-certs\") pod \"maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p\" (UID: \"4dba43a0-1306-49e2-8f2c-bc5e221381d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p"
Nov 03 20:23:26.246925 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:26.246904    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwjfc\" (UniqueName: \"kubernetes.io/projected/4dba43a0-1306-49e2-8f2c-bc5e221381d2-kube-api-access-dwjfc\") pod \"maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p\" (UID: \"4dba43a0-1306-49e2-8f2c-bc5e221381d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p"
Nov 03 20:23:26.260758 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:26.260738    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p"
Nov 03 20:23:26.601293 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:26.601068    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p"]
Nov 03 20:23:26.618820 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:23:26.618746    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dba43a0_1306_49e2_8f2c_bc5e221381d2.slice/crio-c4a64a64dff61a8cd303f08fa4c225887b1e9655cfcec4965ac2ba54ad52a34a WatchSource:0}: Error finding container c4a64a64dff61a8cd303f08fa4c225887b1e9655cfcec4965ac2ba54ad52a34a: Status 404 returned error can't find the container with id c4a64a64dff61a8cd303f08fa4c225887b1e9655cfcec4965ac2ba54ad52a34a
Nov 03 20:23:27.552641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:27.541908    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p" event={"ID":"4dba43a0-1306-49e2-8f2c-bc5e221381d2","Type":"ContainerStarted","Data":"c4a64a64dff61a8cd303f08fa4c225887b1e9655cfcec4965ac2ba54ad52a34a"}
Nov 03 20:23:27.943669 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:27.943638    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z"]
Nov 03 20:23:27.943874 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:27.943849    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z" podUID="e5733b90-6f8c-4105-972f-465f13b91c8a" containerName="maas-api" containerID="cri-o://8aaf06856bbfca477f1ecfc80af762859535787ee3cb73b035150b78ca34cf48" gracePeriod=30
Nov 03 20:23:28.109117 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:28.102622    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z" podUID="e5733b90-6f8c-4105-972f-465f13b91c8a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.128:8443/health\": dial tcp 10.134.0.128:8443: connect: connection refused"
Nov 03 20:23:28.522093 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:28.522067    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z"
Nov 03 20:23:28.555250 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:28.555216    4421 generic.go:356] "Generic (PLEG): container finished" podID="e5733b90-6f8c-4105-972f-465f13b91c8a" containerID="8aaf06856bbfca477f1ecfc80af762859535787ee3cb73b035150b78ca34cf48" exitCode=0
Nov 03 20:23:28.555740 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:28.555290    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z"
Nov 03 20:23:28.555740 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:28.555296    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z" event={"ID":"e5733b90-6f8c-4105-972f-465f13b91c8a","Type":"ContainerDied","Data":"8aaf06856bbfca477f1ecfc80af762859535787ee3cb73b035150b78ca34cf48"}
Nov 03 20:23:28.555740 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:28.555354    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z" event={"ID":"e5733b90-6f8c-4105-972f-465f13b91c8a","Type":"ContainerDied","Data":"4ba42e1af7124a7f3a90579e0781c0509cb6632b6f69d3cf05896a5f9dabe73e"}
Nov 03 20:23:28.555740 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:28.555381    4421 scope.go:122] "RemoveContainer" containerID="8aaf06856bbfca477f1ecfc80af762859535787ee3cb73b035150b78ca34cf48"
Nov 03 20:23:28.557807 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:28.557782    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p" event={"ID":"4dba43a0-1306-49e2-8f2c-bc5e221381d2","Type":"ContainerStarted","Data":"d1fc3c72f0ae388bc87f0a235850bc56aa434ff11fcd0526cf93c579a972b755"}
Nov 03 20:23:28.557969 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:28.557953    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p"
Nov 03 20:23:28.571100 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:28.571079    4421 scope.go:122] "RemoveContainer" containerID="8aaf06856bbfca477f1ecfc80af762859535787ee3cb73b035150b78ca34cf48"
Nov 03 20:23:28.571569 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:23:28.571542    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aaf06856bbfca477f1ecfc80af762859535787ee3cb73b035150b78ca34cf48\": container with ID starting with 8aaf06856bbfca477f1ecfc80af762859535787ee3cb73b035150b78ca34cf48 not found: ID does not exist" containerID="8aaf06856bbfca477f1ecfc80af762859535787ee3cb73b035150b78ca34cf48"
Nov 03 20:23:28.571721 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:28.571579    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aaf06856bbfca477f1ecfc80af762859535787ee3cb73b035150b78ca34cf48"} err="failed to get container status \"8aaf06856bbfca477f1ecfc80af762859535787ee3cb73b035150b78ca34cf48\": rpc error: code = NotFound desc = could not find container \"8aaf06856bbfca477f1ecfc80af762859535787ee3cb73b035150b78ca34cf48\": container with ID starting with 8aaf06856bbfca477f1ecfc80af762859535787ee3cb73b035150b78ca34cf48 not found: ID does not exist"
Nov 03 20:23:28.578483 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:28.578457    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5733b90-6f8c-4105-972f-465f13b91c8a-kube-api-access-qpv4b\" (UniqueName: \"kubernetes.io/projected/e5733b90-6f8c-4105-972f-465f13b91c8a-kube-api-access-qpv4b\") pod \"e5733b90-6f8c-4105-972f-465f13b91c8a\" (UID: \"e5733b90-6f8c-4105-972f-465f13b91c8a\") "
Nov 03 20:23:28.578658 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:28.578575    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5733b90-6f8c-4105-972f-465f13b91c8a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5733b90-6f8c-4105-972f-465f13b91c8a-metrics-certs\") pod \"e5733b90-6f8c-4105-972f-465f13b91c8a\" (UID: \"e5733b90-6f8c-4105-972f-465f13b91c8a\") "
Nov 03 20:23:28.578754 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:28.578735    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5733b90-6f8c-4105-972f-465f13b91c8a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5733b90-6f8c-4105-972f-465f13b91c8a-openshift-service-ca\") pod \"e5733b90-6f8c-4105-972f-465f13b91c8a\" (UID: \"e5733b90-6f8c-4105-972f-465f13b91c8a\") "
Nov 03 20:23:28.578925 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:28.578909    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5733b90-6f8c-4105-972f-465f13b91c8a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5733b90-6f8c-4105-972f-465f13b91c8a-maas-api-tls\") pod \"e5733b90-6f8c-4105-972f-465f13b91c8a\" (UID: \"e5733b90-6f8c-4105-972f-465f13b91c8a\") "
Nov 03 20:23:28.579125 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:28.579092    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5733b90-6f8c-4105-972f-465f13b91c8a-openshift-service-ca" pod "e5733b90-6f8c-4105-972f-465f13b91c8a" (UID: "e5733b90-6f8c-4105-972f-465f13b91c8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:28.579649 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:28.579619    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5733b90-6f8c-4105-972f-465f13b91c8a-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:28.581025 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:28.581003    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5733b90-6f8c-4105-972f-465f13b91c8a-kube-api-access-qpv4b" pod "e5733b90-6f8c-4105-972f-465f13b91c8a" (UID: "e5733b90-6f8c-4105-972f-465f13b91c8a"). InnerVolumeSpecName "kube-api-access-qpv4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:28.581393 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:28.581367    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5733b90-6f8c-4105-972f-465f13b91c8a-maas-api-tls" pod "e5733b90-6f8c-4105-972f-465f13b91c8a" (UID: "e5733b90-6f8c-4105-972f-465f13b91c8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:23:28.582744 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:28.582724    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5733b90-6f8c-4105-972f-465f13b91c8a-metrics-certs" pod "e5733b90-6f8c-4105-972f-465f13b91c8a" (UID: "e5733b90-6f8c-4105-972f-465f13b91c8a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:23:28.593127 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:28.593070    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p" podStartSLOduration=2.615889583 podStartE2EDuration="3.593051281s" totalImagesPullingTime="977.161698ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:28.589792504 +0000 UTC m=+1338.722920966" watchObservedRunningTime="2026-11-03 20:23:28.593051281 +0000 UTC m=+1338.726179744"
Nov 03 20:23:28.681723 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:28.681636    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5733b90-6f8c-4105-972f-465f13b91c8a-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:28.681723 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:28.681672    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpv4b\" (UniqueName: \"kubernetes.io/projected/e5733b90-6f8c-4105-972f-465f13b91c8a-kube-api-access-qpv4b\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:28.681723 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:28.681686    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5733b90-6f8c-4105-972f-465f13b91c8a-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:29.115702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:29.115662    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z"]
Nov 03 20:23:29.171143 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:29.171112    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-xx9xb_79a63610-3e85-4ddd-a7a8-6ceccc2b3f83/payload-processing/0.log"
Nov 03 20:23:29.179939 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:29.179905    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-efba68-56bb8c86dc-6db6z"]
Nov 03 20:23:29.391680 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:29.391006    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-xx9xb_79a63610-3e85-4ddd-a7a8-6ceccc2b3f83/payload-processing/0.log"
Nov 03 20:23:29.591724 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:29.591639    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl_d3fd1d16-f04c-4807-8991-1987e5245fb8/payload-processing/0.log"
Nov 03 20:23:30.023413 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:30.023379    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5733b90-6f8c-4105-972f-465f13b91c8a" path="/var/lib/kubelet/pods/e5733b90-6f8c-4105-972f-465f13b91c8a/volumes"
Nov 03 20:23:34.631628 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:34.626426    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p"
Nov 03 20:23:35.655428 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:35.651781    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r"]
Nov 03 20:23:35.655428 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:35.652896    4421 cpu_manager.go:414] "containerMap: removing container" podUID="e5733b90-6f8c-4105-972f-465f13b91c8a" containerName="maas-api"
Nov 03 20:23:35.655428 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:35.652915    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5733b90-6f8c-4105-972f-465f13b91c8a" containerName="maas-api"
Nov 03 20:23:35.655428 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:35.653093    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5733b90-6f8c-4105-972f-465f13b91c8a" containerName="maas-api"
Nov 03 20:23:35.660492 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:35.660462    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r"
Nov 03 20:23:35.679579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:35.677233    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-9e5029-serving-cert\""
Nov 03 20:23:35.682483 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:35.682437    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r"]
Nov 03 20:23:35.733018 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:35.732982    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvz6k\" (UniqueName: \"kubernetes.io/projected/b319a32a-24a1-4380-aa49-d0ac98924905-kube-api-access-nvz6k\") pod \"maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r\" (UID: \"b319a32a-24a1-4380-aa49-d0ac98924905\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r"
Nov 03 20:23:35.733203 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:35.733078    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b319a32a-24a1-4380-aa49-d0ac98924905-metrics-certs\") pod \"maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r\" (UID: \"b319a32a-24a1-4380-aa49-d0ac98924905\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r"
Nov 03 20:23:35.733203 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:35.733130    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b319a32a-24a1-4380-aa49-d0ac98924905-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r\" (UID: \"b319a32a-24a1-4380-aa49-d0ac98924905\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r"
Nov 03 20:23:35.733399 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:35.733217    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b319a32a-24a1-4380-aa49-d0ac98924905-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r\" (UID: \"b319a32a-24a1-4380-aa49-d0ac98924905\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r"
Nov 03 20:23:35.861398 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:35.858749    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvz6k\" (UniqueName: \"kubernetes.io/projected/b319a32a-24a1-4380-aa49-d0ac98924905-kube-api-access-nvz6k\") pod \"maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r\" (UID: \"b319a32a-24a1-4380-aa49-d0ac98924905\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r"
Nov 03 20:23:35.861398 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:35.858849    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b319a32a-24a1-4380-aa49-d0ac98924905-metrics-certs\") pod \"maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r\" (UID: \"b319a32a-24a1-4380-aa49-d0ac98924905\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r"
Nov 03 20:23:35.861398 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:35.858902    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b319a32a-24a1-4380-aa49-d0ac98924905-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r\" (UID: \"b319a32a-24a1-4380-aa49-d0ac98924905\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r"
Nov 03 20:23:35.861398 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:35.858988    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b319a32a-24a1-4380-aa49-d0ac98924905-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r\" (UID: \"b319a32a-24a1-4380-aa49-d0ac98924905\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r"
Nov 03 20:23:35.865288 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:35.864949    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b319a32a-24a1-4380-aa49-d0ac98924905-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r\" (UID: \"b319a32a-24a1-4380-aa49-d0ac98924905\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r"
Nov 03 20:23:35.865879 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:23:35.865582    4421 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-serving-cert: secret "maas-api-e2e-praxis-b-9e5029-serving-cert" not found
Nov 03 20:23:35.865879 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:23:35.865665    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b319a32a-24a1-4380-aa49-d0ac98924905-maas-api-tls podName:b319a32a-24a1-4380-aa49-d0ac98924905 nodeName:}" failed. No retries permitted until 2026-11-03 20:23:36.36564705 +0000 UTC m=+1346.498775507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b319a32a-24a1-4380-aa49-d0ac98924905-maas-api-tls") pod "maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r" (UID: "b319a32a-24a1-4380-aa49-d0ac98924905") : secret "maas-api-e2e-praxis-b-9e5029-serving-cert" not found
Nov 03 20:23:35.892621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:35.890981    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b319a32a-24a1-4380-aa49-d0ac98924905-metrics-certs\") pod \"maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r\" (UID: \"b319a32a-24a1-4380-aa49-d0ac98924905\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r"
Nov 03 20:23:35.895761 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:35.895676    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvz6k\" (UniqueName: \"kubernetes.io/projected/b319a32a-24a1-4380-aa49-d0ac98924905-kube-api-access-nvz6k\") pod \"maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r\" (UID: \"b319a32a-24a1-4380-aa49-d0ac98924905\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r"
Nov 03 20:23:36.411621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:36.390759    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b319a32a-24a1-4380-aa49-d0ac98924905-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r\" (UID: \"b319a32a-24a1-4380-aa49-d0ac98924905\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r"
Nov 03 20:23:36.411621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:36.411309    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b319a32a-24a1-4380-aa49-d0ac98924905-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r\" (UID: \"b319a32a-24a1-4380-aa49-d0ac98924905\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r"
Nov 03 20:23:36.589819 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:36.589779    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r"
Nov 03 20:23:37.007513 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:36.982382    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r"]
Nov 03 20:23:37.155583 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:37.155541    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r" event={"ID":"b319a32a-24a1-4380-aa49-d0ac98924905","Type":"ContainerStarted","Data":"0bca35ff66bb64b981c914d8df7ca838439faa2cb43c0bac14c1f5276e5c51fb"}
Nov 03 20:23:38.167237 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:38.167193    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r" event={"ID":"b319a32a-24a1-4380-aa49-d0ac98924905","Type":"ContainerStarted","Data":"bebd8712575b2f22ebe6279c8e588f84919a5d2c055bb0386fcacceba39797ed"}
Nov 03 20:23:38.167724 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:38.167415    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r"
Nov 03 20:23:38.189884 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:38.189750    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r" podStartSLOduration=2.698969287 podStartE2EDuration="3.189715766s" totalImagesPullingTime="490.746479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:38.189250741 +0000 UTC m=+1348.322379203" watchObservedRunningTime="2026-11-03 20:23:38.189715766 +0000 UTC m=+1348.322844229"
Nov 03 20:23:41.089958 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:23:41.089923    4421 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@13-23502-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod436959f1_0deb_43cc_b8e0_ca75a6f0afbd.slice/crio-2a9a4ea1e1b460a1696e95d47270d6869143cb09f49d12564b190da7bd639219.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:23:41.194764 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:41.194734    4421 generic.go:356] "Generic (PLEG): container finished" podID="436959f1-0deb-43cc-b8e0-ca75a6f0afbd" containerID="2a9a4ea1e1b460a1696e95d47270d6869143cb09f49d12564b190da7bd639219" exitCode=0
Nov 03 20:23:41.194940 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:41.194807    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq" event={"ID":"436959f1-0deb-43cc-b8e0-ca75a6f0afbd","Type":"ContainerDied","Data":"2a9a4ea1e1b460a1696e95d47270d6869143cb09f49d12564b190da7bd639219"}
Nov 03 20:23:41.195314 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:41.195297    4421 scope.go:122] "RemoveContainer" containerID="2a9a4ea1e1b460a1696e95d47270d6869143cb09f49d12564b190da7bd639219"
Nov 03 20:23:41.196612 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:41.196579    4421 generic.go:356] "Generic (PLEG): container finished" podID="f8732447-9533-40b3-8da9-e8d6d946785e" containerID="73120e7fc615b4e82e5509496d1f0d1f8614ff615bd7e4efc2a949681dbc8f11" exitCode=0
Nov 03 20:23:41.196727 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:41.196672    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9" event={"ID":"f8732447-9533-40b3-8da9-e8d6d946785e","Type":"ContainerDied","Data":"73120e7fc615b4e82e5509496d1f0d1f8614ff615bd7e4efc2a949681dbc8f11"}
Nov 03 20:23:41.197580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:41.197092    4421 scope.go:122] "RemoveContainer" containerID="73120e7fc615b4e82e5509496d1f0d1f8614ff615bd7e4efc2a949681dbc8f11"
Nov 03 20:23:41.197580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:41.197478    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:23:41.197580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:41.197553    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:23:41.197784 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:41.197591    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:23:41.198926 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:41.198897    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:23:41.199042 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:41.198973    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:23:41.199042 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:41.199018    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:23:42.236995 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:42.236950    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9" event={"ID":"f8732447-9533-40b3-8da9-e8d6d946785e","Type":"ContainerStarted","Data":"0e9809d2925adf5d53a6e861bfc037814ecb8726578b55dde3128a83080d87aa"}
Nov 03 20:23:42.262196 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:42.262152    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq" event={"ID":"436959f1-0deb-43cc-b8e0-ca75a6f0afbd","Type":"ContainerStarted","Data":"ec18ac3db0893853f2e1841f582006c0a793ff4c6d5927261fa1cc5faafe8f24"}
Nov 03 20:23:42.828121 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:42.828083    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:23:42.829418 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:42.829393    4421 patch_prober.go:29] interesting pod/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 20:23:42.829548 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:42.829440    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq" podUID="436959f1-0deb-43cc-b8e0-ca75a6f0afbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 20:23:43.143668 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:43.134350    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:23:43.143668 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:43.134489    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 03 20:23:43.143668 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:43.134532    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9" podUID="f8732447-9533-40b3-8da9-e8d6d946785e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 03 20:23:43.828207 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:43.828170    4421 patch_prober.go:29] interesting pod/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 20:23:43.828689 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:43.828235    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq" podUID="436959f1-0deb-43cc-b8e0-ca75a6f0afbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 20:23:44.134231 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:44.134155    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 03 20:23:44.134231 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:44.134208    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9" podUID="f8732447-9533-40b3-8da9-e8d6d946785e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 03 20:23:44.204171 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:44.204139    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r"
Nov 03 20:23:44.828130 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:44.828097    4421 patch_prober.go:29] interesting pod/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 20:23:44.828315 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:44.828148    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq" podUID="436959f1-0deb-43cc-b8e0-ca75a6f0afbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 20:23:45.133930 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:45.133807    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 03 20:23:45.133930 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:45.133856    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9" podUID="f8732447-9533-40b3-8da9-e8d6d946785e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 03 20:23:45.836830 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:45.836727    4421 patch_prober.go:29] interesting pod/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 20:23:45.837424 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:45.837394    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq" podUID="436959f1-0deb-43cc-b8e0-ca75a6f0afbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 20:23:46.133687 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:46.133563    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 03 20:23:46.133687 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:46.133666    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9" podUID="f8732447-9533-40b3-8da9-e8d6d946785e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 03 20:23:46.311054 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:46.311022    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:23:46.827992 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:46.827959    4421 patch_prober.go:29] interesting pod/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 20:23:46.828214 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:46.828014    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq" podUID="436959f1-0deb-43cc-b8e0-ca75a6f0afbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 20:23:47.134240 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:47.134162    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 03 20:23:47.134240 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:47.134211    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9" podUID="f8732447-9533-40b3-8da9-e8d6d946785e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 03 20:23:47.828829 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:47.828794    4421 patch_prober.go:29] interesting pod/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 20:23:47.829013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:47.828847    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq" podUID="436959f1-0deb-43cc-b8e0-ca75a6f0afbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 20:23:48.169827 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:48.169720    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:23:48.170275 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:48.170255    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:23:48.171417 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:48.171393    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:23:48.546927 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:48.546893    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn"]
Nov 03 20:23:48.547171 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:48.547143    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn" podUID="e81ae3d3-e34d-4c9b-8f0b-739204ede414" containerName="maas-api" containerID="cri-o://350c013f72ae603424a12c6d2cafaceba8b72fdd6a2202ae95866df237f1ed79" gracePeriod=30
Nov 03 20:23:48.913674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:48.910906    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:23:49.011627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:49.002185    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:23:49.154940 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:49.154912    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn"
Nov 03 20:23:49.332425 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:49.331360    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e81ae3d3-e34d-4c9b-8f0b-739204ede414-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e81ae3d3-e34d-4c9b-8f0b-739204ede414-openshift-service-ca\") pod \"e81ae3d3-e34d-4c9b-8f0b-739204ede414\" (UID: \"e81ae3d3-e34d-4c9b-8f0b-739204ede414\") "
Nov 03 20:23:49.332425 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:49.331522    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e81ae3d3-e34d-4c9b-8f0b-739204ede414-kube-api-access-w77fr\" (UniqueName: \"kubernetes.io/projected/e81ae3d3-e34d-4c9b-8f0b-739204ede414-kube-api-access-w77fr\") pod \"e81ae3d3-e34d-4c9b-8f0b-739204ede414\" (UID: \"e81ae3d3-e34d-4c9b-8f0b-739204ede414\") "
Nov 03 20:23:49.332425 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:49.331687    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e81ae3d3-e34d-4c9b-8f0b-739204ede414-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e81ae3d3-e34d-4c9b-8f0b-739204ede414-maas-api-tls\") pod \"e81ae3d3-e34d-4c9b-8f0b-739204ede414\" (UID: \"e81ae3d3-e34d-4c9b-8f0b-739204ede414\") "
Nov 03 20:23:49.332425 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:49.331841    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e81ae3d3-e34d-4c9b-8f0b-739204ede414-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e81ae3d3-e34d-4c9b-8f0b-739204ede414-metrics-certs\") pod \"e81ae3d3-e34d-4c9b-8f0b-739204ede414\" (UID: \"e81ae3d3-e34d-4c9b-8f0b-739204ede414\") "
Nov 03 20:23:49.340013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:49.339982    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e81ae3d3-e34d-4c9b-8f0b-739204ede414-openshift-service-ca" pod "e81ae3d3-e34d-4c9b-8f0b-739204ede414" (UID: "e81ae3d3-e34d-4c9b-8f0b-739204ede414"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:49.344429 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:49.344297    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e81ae3d3-e34d-4c9b-8f0b-739204ede414-maas-api-tls" pod "e81ae3d3-e34d-4c9b-8f0b-739204ede414" (UID: "e81ae3d3-e34d-4c9b-8f0b-739204ede414"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:23:49.383589 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:49.383553    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e81ae3d3-e34d-4c9b-8f0b-739204ede414-kube-api-access-w77fr" pod "e81ae3d3-e34d-4c9b-8f0b-739204ede414" (UID: "e81ae3d3-e34d-4c9b-8f0b-739204ede414"). InnerVolumeSpecName "kube-api-access-w77fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:49.413658 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:49.412425    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e81ae3d3-e34d-4c9b-8f0b-739204ede414-metrics-certs" pod "e81ae3d3-e34d-4c9b-8f0b-739204ede414" (UID: "e81ae3d3-e34d-4c9b-8f0b-739204ede414"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:23:49.425699 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:49.421609    4421 generic.go:356] "Generic (PLEG): container finished" podID="e81ae3d3-e34d-4c9b-8f0b-739204ede414" containerID="350c013f72ae603424a12c6d2cafaceba8b72fdd6a2202ae95866df237f1ed79" exitCode=0
Nov 03 20:23:49.425699 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:49.422910    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn"
Nov 03 20:23:49.425699 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:49.423866    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn" event={"ID":"e81ae3d3-e34d-4c9b-8f0b-739204ede414","Type":"ContainerDied","Data":"350c013f72ae603424a12c6d2cafaceba8b72fdd6a2202ae95866df237f1ed79"}
Nov 03 20:23:49.425699 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:49.424018    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn" event={"ID":"e81ae3d3-e34d-4c9b-8f0b-739204ede414","Type":"ContainerDied","Data":"d8bf4400d9d10b81831840322b5ccb8f068630e46dbad680c6ea2959c0e0c8b9"}
Nov 03 20:23:49.428948 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:49.428491    4421 scope.go:122] "RemoveContainer" containerID="350c013f72ae603424a12c6d2cafaceba8b72fdd6a2202ae95866df237f1ed79"
Nov 03 20:23:49.433777 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:49.433704    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w77fr\" (UniqueName: \"kubernetes.io/projected/e81ae3d3-e34d-4c9b-8f0b-739204ede414-kube-api-access-w77fr\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:49.433777 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:49.433726    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e81ae3d3-e34d-4c9b-8f0b-739204ede414-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:49.433777 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:49.433736    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e81ae3d3-e34d-4c9b-8f0b-739204ede414-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:49.433777 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:49.433745    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e81ae3d3-e34d-4c9b-8f0b-739204ede414-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:49.444256 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:49.444203    4421 scope.go:122] "RemoveContainer" containerID="350c013f72ae603424a12c6d2cafaceba8b72fdd6a2202ae95866df237f1ed79"
Nov 03 20:23:49.444936 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:23:49.444914    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"350c013f72ae603424a12c6d2cafaceba8b72fdd6a2202ae95866df237f1ed79\": container with ID starting with 350c013f72ae603424a12c6d2cafaceba8b72fdd6a2202ae95866df237f1ed79 not found: ID does not exist" containerID="350c013f72ae603424a12c6d2cafaceba8b72fdd6a2202ae95866df237f1ed79"
Nov 03 20:23:49.445021 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:49.444944    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"350c013f72ae603424a12c6d2cafaceba8b72fdd6a2202ae95866df237f1ed79"} err="failed to get container status \"350c013f72ae603424a12c6d2cafaceba8b72fdd6a2202ae95866df237f1ed79\": rpc error: code = NotFound desc = could not find container \"350c013f72ae603424a12c6d2cafaceba8b72fdd6a2202ae95866df237f1ed79\": container with ID starting with 350c013f72ae603424a12c6d2cafaceba8b72fdd6a2202ae95866df237f1ed79 not found: ID does not exist"
Nov 03 20:23:49.460707 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:49.460677    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn"]
Nov 03 20:23:49.464436 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:49.464400    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-82c116-55ccfc947-mrgvn"]
Nov 03 20:23:49.630021 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:23:49.629941    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:23:50.026749 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:50.026710    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e81ae3d3-e34d-4c9b-8f0b-739204ede414" path="/var/lib/kubelet/pods/e81ae3d3-e34d-4c9b-8f0b-739204ede414/volumes"
Nov 03 20:23:50.065282 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:50.065247    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"]
Nov 03 20:23:50.065528 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:50.065503    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn" podUID="c263b0e4-5dd7-4a46-a527-f461b779e42f" containerName="istio-proxy" containerID="cri-o://209fa2c38f7bf6ea798eb4967effe419aa32361835bfaffbb944c41e72897d87" gracePeriod=30
Nov 03 20:23:50.276627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:50.275995    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"]
Nov 03 20:23:50.276627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:50.276233    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4" podUID="6b34f6ef-a056-4dbb-9e6f-d61c78d770b6" containerName="istio-proxy" containerID="cri-o://64b6998a90bc28dcd29cac55c072d91189e307ab590b427480be9360ad220d7c" gracePeriod=30
Nov 03 20:23:51.057625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:51.053659    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25"]
Nov 03 20:23:51.057625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:51.054834    4421 cpu_manager.go:414] "containerMap: removing container" podUID="e81ae3d3-e34d-4c9b-8f0b-739204ede414" containerName="maas-api"
Nov 03 20:23:51.057625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:51.054854    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e81ae3d3-e34d-4c9b-8f0b-739204ede414" containerName="maas-api"
Nov 03 20:23:51.057625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:51.055080    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e81ae3d3-e34d-4c9b-8f0b-739204ede414" containerName="maas-api"
Nov 03 20:23:51.106640 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:51.105618    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25"
Nov 03 20:23:51.191564 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:51.109447    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/675e46aa-b032-4274-90ba-9fdf30de5d57-maas-api-tls\") pod \"maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25\" (UID: \"675e46aa-b032-4274-90ba-9fdf30de5d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25"
Nov 03 20:23:51.227998 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:51.227964    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8rxl\" (UniqueName: \"kubernetes.io/projected/675e46aa-b032-4274-90ba-9fdf30de5d57-kube-api-access-p8rxl\") pod \"maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25\" (UID: \"675e46aa-b032-4274-90ba-9fdf30de5d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25"
Nov 03 20:23:51.228538 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:51.228514    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/675e46aa-b032-4274-90ba-9fdf30de5d57-metrics-certs\") pod \"maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25\" (UID: \"675e46aa-b032-4274-90ba-9fdf30de5d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25"
Nov 03 20:23:51.257804 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:51.252764    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/675e46aa-b032-4274-90ba-9fdf30de5d57-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25\" (UID: \"675e46aa-b032-4274-90ba-9fdf30de5d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25"
Nov 03 20:23:51.266631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:51.113052    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-019ef4-serving-cert\""
Nov 03 20:23:51.273147 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:51.273117    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25"]
Nov 03 20:23:51.369025 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:51.368188    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/675e46aa-b032-4274-90ba-9fdf30de5d57-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25\" (UID: \"675e46aa-b032-4274-90ba-9fdf30de5d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25"
Nov 03 20:23:51.369025 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:51.368407    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/675e46aa-b032-4274-90ba-9fdf30de5d57-maas-api-tls\") pod \"maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25\" (UID: \"675e46aa-b032-4274-90ba-9fdf30de5d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25"
Nov 03 20:23:51.369025 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:51.368438    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8rxl\" (UniqueName: \"kubernetes.io/projected/675e46aa-b032-4274-90ba-9fdf30de5d57-kube-api-access-p8rxl\") pod \"maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25\" (UID: \"675e46aa-b032-4274-90ba-9fdf30de5d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25"
Nov 03 20:23:51.369025 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:51.368486    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/675e46aa-b032-4274-90ba-9fdf30de5d57-metrics-certs\") pod \"maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25\" (UID: \"675e46aa-b032-4274-90ba-9fdf30de5d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25"
Nov 03 20:23:51.369807 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:23:51.369471    4421 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-serving-cert: secret "maas-api-e2e-praxis-a-019ef4-serving-cert" not found
Nov 03 20:23:51.369807 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:23:51.369548    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/675e46aa-b032-4274-90ba-9fdf30de5d57-maas-api-tls podName:675e46aa-b032-4274-90ba-9fdf30de5d57 nodeName:}" failed. No retries permitted until 2026-11-03 20:23:51.869528034 +0000 UTC m=+1362.002656480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/675e46aa-b032-4274-90ba-9fdf30de5d57-maas-api-tls") pod "maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25" (UID: "675e46aa-b032-4274-90ba-9fdf30de5d57") : secret "maas-api-e2e-praxis-a-019ef4-serving-cert" not found
Nov 03 20:23:51.370141 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:51.370123    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/675e46aa-b032-4274-90ba-9fdf30de5d57-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25\" (UID: \"675e46aa-b032-4274-90ba-9fdf30de5d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25"
Nov 03 20:23:51.390280 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:51.382161    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/675e46aa-b032-4274-90ba-9fdf30de5d57-metrics-certs\") pod \"maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25\" (UID: \"675e46aa-b032-4274-90ba-9fdf30de5d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25"
Nov 03 20:23:51.390280 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:51.386514    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8rxl\" (UniqueName: \"kubernetes.io/projected/675e46aa-b032-4274-90ba-9fdf30de5d57-kube-api-access-p8rxl\") pod \"maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25\" (UID: \"675e46aa-b032-4274-90ba-9fdf30de5d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25"
Nov 03 20:23:51.877656 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:51.877624    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/675e46aa-b032-4274-90ba-9fdf30de5d57-maas-api-tls\") pod \"maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25\" (UID: \"675e46aa-b032-4274-90ba-9fdf30de5d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25"
Nov 03 20:23:51.880323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:51.880299    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/675e46aa-b032-4274-90ba-9fdf30de5d57-maas-api-tls\") pod \"maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25\" (UID: \"675e46aa-b032-4274-90ba-9fdf30de5d57\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25"
Nov 03 20:23:51.971241 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:51.971208    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25"
Nov 03 20:23:52.278190 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:52.278155    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25"]
Nov 03 20:23:52.542835 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:52.542735    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25" event={"ID":"675e46aa-b032-4274-90ba-9fdf30de5d57","Type":"ContainerStarted","Data":"e651a6dfcf624394cb6447eb9939616a5cb94d8393ff9adf62368af025f7e371"}
Nov 03 20:23:53.551930 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:53.551880    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25" event={"ID":"675e46aa-b032-4274-90ba-9fdf30de5d57","Type":"ContainerStarted","Data":"bda6278d37d89201e36d2a8984b85008c21d807e4a08d2627d7923fdcaf66b50"}
Nov 03 20:23:53.552376 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:53.552139    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25"
Nov 03 20:23:53.579618 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:53.579552    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25" podStartSLOduration=3.096314031 podStartE2EDuration="3.579541206s" totalImagesPullingTime="483.227175ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:53.577448797 +0000 UTC m=+1363.710577260" watchObservedRunningTime="2026-11-03 20:23:53.579541206 +0000 UTC m=+1363.712669668"
Nov 03 20:23:55.348461 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.348436    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:23:55.492526 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.492496    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-envoy\") pod \"c263b0e4-5dd7-4a46-a527-f461b779e42f\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") "
Nov 03 20:23:55.492719 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.492639    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-podinfo\") pod \"c263b0e4-5dd7-4a46-a527-f461b779e42f\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") "
Nov 03 20:23:55.492800 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.492769    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-data\") pod \"c263b0e4-5dd7-4a46-a527-f461b779e42f\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") "
Nov 03 20:23:55.492863 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.492821    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-token\" (UniqueName: \"kubernetes.io/projected/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-token\") pod \"c263b0e4-5dd7-4a46-a527-f461b779e42f\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") "
Nov 03 20:23:55.492932 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.492916    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c263b0e4-5dd7-4a46-a527-f461b779e42f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c263b0e4-5dd7-4a46-a527-f461b779e42f-istiod-ca-cert\") pod \"c263b0e4-5dd7-4a46-a527-f461b779e42f\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") "
Nov 03 20:23:55.493006 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.492992    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-workload-certs\") pod \"c263b0e4-5dd7-4a46-a527-f461b779e42f\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") "
Nov 03 20:23:55.493066 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.493046    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-workload-socket\") pod \"c263b0e4-5dd7-4a46-a527-f461b779e42f\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") "
Nov 03 20:23:55.493130 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.493118    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-credential-socket\") pod \"c263b0e4-5dd7-4a46-a527-f461b779e42f\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") "
Nov 03 20:23:55.493184 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.493160    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c263b0e4-5dd7-4a46-a527-f461b779e42f-kube-api-access-pbc4f\" (UniqueName: \"kubernetes.io/projected/c263b0e4-5dd7-4a46-a527-f461b779e42f-kube-api-access-pbc4f\") pod \"c263b0e4-5dd7-4a46-a527-f461b779e42f\" (UID: \"c263b0e4-5dd7-4a46-a527-f461b779e42f\") "
Nov 03 20:23:55.493628 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.493574    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-workload-certs" pod "c263b0e4-5dd7-4a46-a527-f461b779e42f" (UID: "c263b0e4-5dd7-4a46-a527-f461b779e42f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:55.493962 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.493940    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:55.494344 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.494313    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c263b0e4-5dd7-4a46-a527-f461b779e42f-istiod-ca-cert" pod "c263b0e4-5dd7-4a46-a527-f461b779e42f" (UID: "c263b0e4-5dd7-4a46-a527-f461b779e42f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:55.494498 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.494474    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-data" pod "c263b0e4-5dd7-4a46-a527-f461b779e42f" (UID: "c263b0e4-5dd7-4a46-a527-f461b779e42f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:55.494587 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.494545    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-credential-socket" pod "c263b0e4-5dd7-4a46-a527-f461b779e42f" (UID: "c263b0e4-5dd7-4a46-a527-f461b779e42f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:55.494587 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.493513    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-workload-socket" pod "c263b0e4-5dd7-4a46-a527-f461b779e42f" (UID: "c263b0e4-5dd7-4a46-a527-f461b779e42f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:55.497108 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.497070    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-podinfo" pod "c263b0e4-5dd7-4a46-a527-f461b779e42f" (UID: "c263b0e4-5dd7-4a46-a527-f461b779e42f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:23:55.497798 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.497769    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-token" pod "c263b0e4-5dd7-4a46-a527-f461b779e42f" (UID: "c263b0e4-5dd7-4a46-a527-f461b779e42f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:55.498812 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.498782    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-envoy" pod "c263b0e4-5dd7-4a46-a527-f461b779e42f" (UID: "c263b0e4-5dd7-4a46-a527-f461b779e42f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:55.498903 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.498885    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c263b0e4-5dd7-4a46-a527-f461b779e42f-kube-api-access-pbc4f" pod "c263b0e4-5dd7-4a46-a527-f461b779e42f" (UID: "c263b0e4-5dd7-4a46-a527-f461b779e42f"). InnerVolumeSpecName "kube-api-access-pbc4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:55.548729 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.548707    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:23:55.568822 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.568796    4421 generic.go:356] "Generic (PLEG): container finished" podID="6b34f6ef-a056-4dbb-9e6f-d61c78d770b6" containerID="64b6998a90bc28dcd29cac55c072d91189e307ab590b427480be9360ad220d7c" exitCode=0
Nov 03 20:23:55.568942 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.568868    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"
Nov 03 20:23:55.568942 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.568889    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4" event={"ID":"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6","Type":"ContainerDied","Data":"64b6998a90bc28dcd29cac55c072d91189e307ab590b427480be9360ad220d7c"}
Nov 03 20:23:55.569054 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.568954    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4" event={"ID":"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6","Type":"ContainerDied","Data":"11437d2ec6d9f8b10414028bd1525a6d17ceccf661129c117a55ce76c5028e65"}
Nov 03 20:23:55.569054 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.568985    4421 scope.go:122] "RemoveContainer" containerID="64b6998a90bc28dcd29cac55c072d91189e307ab590b427480be9360ad220d7c"
Nov 03 20:23:55.571187 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.571160    4421 generic.go:356] "Generic (PLEG): container finished" podID="c263b0e4-5dd7-4a46-a527-f461b779e42f" containerID="209fa2c38f7bf6ea798eb4967effe419aa32361835bfaffbb944c41e72897d87" exitCode=0
Nov 03 20:23:55.571291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.571201    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"
Nov 03 20:23:55.571366 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.571278    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn" event={"ID":"c263b0e4-5dd7-4a46-a527-f461b779e42f","Type":"ContainerDied","Data":"209fa2c38f7bf6ea798eb4967effe419aa32361835bfaffbb944c41e72897d87"}
Nov 03 20:23:55.571366 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.571352    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn" event={"ID":"c263b0e4-5dd7-4a46-a527-f461b779e42f","Type":"ContainerDied","Data":"28efa2e5042101fd3a800c66a3b7829da664fca2e36bff0b23027aed75120707"}
Nov 03 20:23:55.599870 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.594051    4421 scope.go:122] "RemoveContainer" containerID="64b6998a90bc28dcd29cac55c072d91189e307ab590b427480be9360ad220d7c"
Nov 03 20:23:55.599870 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.598269    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:55.599870 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.598295    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c263b0e4-5dd7-4a46-a527-f461b779e42f-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:55.599870 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.598310    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:55.599870 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.598324    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:55.599870 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.598338    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbc4f\" (UniqueName: \"kubernetes.io/projected/c263b0e4-5dd7-4a46-a527-f461b779e42f-kube-api-access-pbc4f\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:55.599870 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.598353    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:55.599870 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.598366    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:55.599870 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.598379    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c263b0e4-5dd7-4a46-a527-f461b779e42f-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:55.602791 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:23:55.602765    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64b6998a90bc28dcd29cac55c072d91189e307ab590b427480be9360ad220d7c\": container with ID starting with 64b6998a90bc28dcd29cac55c072d91189e307ab590b427480be9360ad220d7c not found: ID does not exist" containerID="64b6998a90bc28dcd29cac55c072d91189e307ab590b427480be9360ad220d7c"
Nov 03 20:23:55.603271 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.602802    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b6998a90bc28dcd29cac55c072d91189e307ab590b427480be9360ad220d7c"} err="failed to get container status \"64b6998a90bc28dcd29cac55c072d91189e307ab590b427480be9360ad220d7c\": rpc error: code = NotFound desc = could not find container \"64b6998a90bc28dcd29cac55c072d91189e307ab590b427480be9360ad220d7c\": container with ID starting with 64b6998a90bc28dcd29cac55c072d91189e307ab590b427480be9360ad220d7c not found: ID does not exist"
Nov 03 20:23:55.603271 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.602827    4421 scope.go:122] "RemoveContainer" containerID="209fa2c38f7bf6ea798eb4967effe419aa32361835bfaffbb944c41e72897d87"
Nov 03 20:23:55.609401 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.609376    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6"]
Nov 03 20:23:55.616925 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.616900    4421 scope.go:122] "RemoveContainer" containerID="01a6879065dc2564a3816015282a6cd62df4f2f7a9817ab678edc8184ca4a60e"
Nov 03 20:23:55.632718 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.624463    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj"]
Nov 03 20:23:55.632718 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.631900    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"]
Nov 03 20:23:55.645227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.645199    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-vk6dn"]
Nov 03 20:23:55.646532 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.646411    4421 scope.go:122] "RemoveContainer" containerID="209fa2c38f7bf6ea798eb4967effe419aa32361835bfaffbb944c41e72897d87"
Nov 03 20:23:55.650405 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:23:55.647214    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"209fa2c38f7bf6ea798eb4967effe419aa32361835bfaffbb944c41e72897d87\": container with ID starting with 209fa2c38f7bf6ea798eb4967effe419aa32361835bfaffbb944c41e72897d87 not found: ID does not exist" containerID="209fa2c38f7bf6ea798eb4967effe419aa32361835bfaffbb944c41e72897d87"
Nov 03 20:23:55.650405 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.647244    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"209fa2c38f7bf6ea798eb4967effe419aa32361835bfaffbb944c41e72897d87"} err="failed to get container status \"209fa2c38f7bf6ea798eb4967effe419aa32361835bfaffbb944c41e72897d87\": rpc error: code = NotFound desc = could not find container \"209fa2c38f7bf6ea798eb4967effe419aa32361835bfaffbb944c41e72897d87\": container with ID starting with 209fa2c38f7bf6ea798eb4967effe419aa32361835bfaffbb944c41e72897d87 not found: ID does not exist"
Nov 03 20:23:55.650405 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.647266    4421 scope.go:122] "RemoveContainer" containerID="01a6879065dc2564a3816015282a6cd62df4f2f7a9817ab678edc8184ca4a60e"
Nov 03 20:23:55.650405 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:23:55.647554    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01a6879065dc2564a3816015282a6cd62df4f2f7a9817ab678edc8184ca4a60e\": container with ID starting with 01a6879065dc2564a3816015282a6cd62df4f2f7a9817ab678edc8184ca4a60e not found: ID does not exist" containerID="01a6879065dc2564a3816015282a6cd62df4f2f7a9817ab678edc8184ca4a60e"
Nov 03 20:23:55.650405 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.647579    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01a6879065dc2564a3816015282a6cd62df4f2f7a9817ab678edc8184ca4a60e"} err="failed to get container status \"01a6879065dc2564a3816015282a6cd62df4f2f7a9817ab678edc8184ca4a60e\": rpc error: code = NotFound desc = could not find container \"01a6879065dc2564a3816015282a6cd62df4f2f7a9817ab678edc8184ca4a60e\": container with ID starting with 01a6879065dc2564a3816015282a6cd62df4f2f7a9817ab678edc8184ca4a60e not found: ID does not exist"
Nov 03 20:23:55.699464 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.699435    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-kube-api-access-g8sfl\" (UniqueName: \"kubernetes.io/projected/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-kube-api-access-g8sfl\") pod \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") "
Nov 03 20:23:55.699697 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.699473    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-workload-certs\") pod \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") "
Nov 03 20:23:55.699697 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.699537    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-workload-socket\") pod \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") "
Nov 03 20:23:55.699697 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.699614    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-credential-socket\") pod \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") "
Nov 03 20:23:55.699888 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.699729    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-data\") pod \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") "
Nov 03 20:23:55.699888 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.699810    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-workload-certs" pod "6b34f6ef-a056-4dbb-9e6f-d61c78d770b6" (UID: "6b34f6ef-a056-4dbb-9e6f-d61c78d770b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:55.699888 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.699825    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-envoy\") pod \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") "
Nov 03 20:23:55.699888 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.699854    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-token\" (UniqueName: \"kubernetes.io/projected/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-token\") pod \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") "
Nov 03 20:23:55.700103 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.699974    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-podinfo\") pod \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") "
Nov 03 20:23:55.700103 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.700012    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-credential-socket" pod "6b34f6ef-a056-4dbb-9e6f-d61c78d770b6" (UID: "6b34f6ef-a056-4dbb-9e6f-d61c78d770b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:55.700103 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.700035    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istiod-ca-cert\") pod \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\" (UID: \"6b34f6ef-a056-4dbb-9e6f-d61c78d770b6\") "
Nov 03 20:23:55.700303 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.700173    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-workload-socket" pod "6b34f6ef-a056-4dbb-9e6f-d61c78d770b6" (UID: "6b34f6ef-a056-4dbb-9e6f-d61c78d770b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:55.700746 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.700696    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-data" pod "6b34f6ef-a056-4dbb-9e6f-d61c78d770b6" (UID: "6b34f6ef-a056-4dbb-9e6f-d61c78d770b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:55.700899 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.700775    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:55.700899 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.700796    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:55.700899 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.700810    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:55.701137 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.701029    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istiod-ca-cert" pod "6b34f6ef-a056-4dbb-9e6f-d61c78d770b6" (UID: "6b34f6ef-a056-4dbb-9e6f-d61c78d770b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:23:55.702831 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.702800    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-envoy" pod "6b34f6ef-a056-4dbb-9e6f-d61c78d770b6" (UID: "6b34f6ef-a056-4dbb-9e6f-d61c78d770b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:23:55.703758 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.703727    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-kube-api-access-g8sfl" pod "6b34f6ef-a056-4dbb-9e6f-d61c78d770b6" (UID: "6b34f6ef-a056-4dbb-9e6f-d61c78d770b6"). InnerVolumeSpecName "kube-api-access-g8sfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:55.704015 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.703924    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-token" pod "6b34f6ef-a056-4dbb-9e6f-d61c78d770b6" (UID: "6b34f6ef-a056-4dbb-9e6f-d61c78d770b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:23:55.705217 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.705191    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-podinfo" pod "6b34f6ef-a056-4dbb-9e6f-d61c78d770b6" (UID: "6b34f6ef-a056-4dbb-9e6f-d61c78d770b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:23:55.802003 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.801959    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:55.802003 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.801999    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:55.802003 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.802008    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:55.802244 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.802017    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:55.802244 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.802026    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:55.802244 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.802035    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8sfl\" (UniqueName: \"kubernetes.io/projected/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6-kube-api-access-g8sfl\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:23:55.899340 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.899307    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"]
Nov 03 20:23:55.903508 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:55.903482    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-vzhd4"]
Nov 03 20:23:56.019110 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:56.019077    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b34f6ef-a056-4dbb-9e6f-d61c78d770b6" path="/var/lib/kubelet/pods/6b34f6ef-a056-4dbb-9e6f-d61c78d770b6/volumes"
Nov 03 20:23:56.019761 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:56.019735    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c263b0e4-5dd7-4a46-a527-f461b779e42f" path="/var/lib/kubelet/pods/c263b0e4-5dd7-4a46-a527-f461b779e42f/volumes"
Nov 03 20:23:57.538626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:57.530890    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn"]
Nov 03 20:23:57.858677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:57.858505    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j"]
Nov 03 20:23:59.563935 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:59.563903    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv"]
Nov 03 20:23:59.564415 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:59.564163    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv" podUID="f479675b-17b7-432b-aef4-9cbc3bfb7359" containerName="maas-api" containerID="cri-o://23b5c1007d4b26ed38f71376dc5e8e21b66eaca3256c8a1b326ae3ba4431b74c" gracePeriod=30
Nov 03 20:23:59.569469 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:59.569445    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25"
Nov 03 20:23:59.992516 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:59.990982    4421 generic.go:356] "Generic (PLEG): container finished" podID="f479675b-17b7-432b-aef4-9cbc3bfb7359" containerID="23b5c1007d4b26ed38f71376dc5e8e21b66eaca3256c8a1b326ae3ba4431b74c" exitCode=0
Nov 03 20:23:59.992516 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:59.991084    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv" event={"ID":"f479675b-17b7-432b-aef4-9cbc3bfb7359","Type":"ContainerDied","Data":"23b5c1007d4b26ed38f71376dc5e8e21b66eaca3256c8a1b326ae3ba4431b74c"}
Nov 03 20:23:59.992516 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:59.991139    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv" event={"ID":"f479675b-17b7-432b-aef4-9cbc3bfb7359","Type":"ContainerDied","Data":"a194613d02dea7a4372d3a1eb13a45c1ca418091e602eda0c0f57291f604bdf4"}
Nov 03 20:23:59.992516 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:23:59.991163    4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a194613d02dea7a4372d3a1eb13a45c1ca418091e602eda0c0f57291f604bdf4"
Nov 03 20:24:00.073815 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:00.073760    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv"
Nov 03 20:24:00.166534 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:00.166456    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f479675b-17b7-432b-aef4-9cbc3bfb7359-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f479675b-17b7-432b-aef4-9cbc3bfb7359-metrics-certs\") pod \"f479675b-17b7-432b-aef4-9cbc3bfb7359\" (UID: \"f479675b-17b7-432b-aef4-9cbc3bfb7359\") "
Nov 03 20:24:00.166700 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:00.166673    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f479675b-17b7-432b-aef4-9cbc3bfb7359-kube-api-access-4xxw6\" (UniqueName: \"kubernetes.io/projected/f479675b-17b7-432b-aef4-9cbc3bfb7359-kube-api-access-4xxw6\") pod \"f479675b-17b7-432b-aef4-9cbc3bfb7359\" (UID: \"f479675b-17b7-432b-aef4-9cbc3bfb7359\") "
Nov 03 20:24:00.166755 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:00.166733    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f479675b-17b7-432b-aef4-9cbc3bfb7359-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f479675b-17b7-432b-aef4-9cbc3bfb7359-maas-api-tls\") pod \"f479675b-17b7-432b-aef4-9cbc3bfb7359\" (UID: \"f479675b-17b7-432b-aef4-9cbc3bfb7359\") "
Nov 03 20:24:00.166820 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:00.166794    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f479675b-17b7-432b-aef4-9cbc3bfb7359-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f479675b-17b7-432b-aef4-9cbc3bfb7359-openshift-service-ca\") pod \"f479675b-17b7-432b-aef4-9cbc3bfb7359\" (UID: \"f479675b-17b7-432b-aef4-9cbc3bfb7359\") "
Nov 03 20:24:00.167192 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:00.167167    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f479675b-17b7-432b-aef4-9cbc3bfb7359-openshift-service-ca" pod "f479675b-17b7-432b-aef4-9cbc3bfb7359" (UID: "f479675b-17b7-432b-aef4-9cbc3bfb7359"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:24:00.167799 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:00.167775    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f479675b-17b7-432b-aef4-9cbc3bfb7359-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:00.168813 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:00.168791    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f479675b-17b7-432b-aef4-9cbc3bfb7359-metrics-certs" pod "f479675b-17b7-432b-aef4-9cbc3bfb7359" (UID: "f479675b-17b7-432b-aef4-9cbc3bfb7359"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:24:00.168943 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:00.168929    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f479675b-17b7-432b-aef4-9cbc3bfb7359-kube-api-access-4xxw6" pod "f479675b-17b7-432b-aef4-9cbc3bfb7359" (UID: "f479675b-17b7-432b-aef4-9cbc3bfb7359"). InnerVolumeSpecName "kube-api-access-4xxw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:24:00.169055 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:00.169043    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f479675b-17b7-432b-aef4-9cbc3bfb7359-maas-api-tls" pod "f479675b-17b7-432b-aef4-9cbc3bfb7359" (UID: "f479675b-17b7-432b-aef4-9cbc3bfb7359"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:24:00.269432 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:00.269184    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f479675b-17b7-432b-aef4-9cbc3bfb7359-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:00.269432 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:00.269219    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xxw6\" (UniqueName: \"kubernetes.io/projected/f479675b-17b7-432b-aef4-9cbc3bfb7359-kube-api-access-4xxw6\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:00.269432 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:00.269234    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f479675b-17b7-432b-aef4-9cbc3bfb7359-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:00.610343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:00.610306    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6" podUID="6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8" containerName="payload-processing" containerID="cri-o://6dc54b7e81966bf70991beb151eca12e307b88340845ae711960b12a3c3fb230" gracePeriod=25
Nov 03 20:24:00.625827 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:00.625794    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj" podUID="b3dbca1e-d679-4e9e-960d-9006b92b6d72" containerName="payload-pre-processing" containerID="cri-o://27e621c3ee88618bf51812cd8b1c6758ee75a93d615adb282921e5193946a459" gracePeriod=25
Nov 03 20:24:00.916739 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:00.916709    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6"
Nov 03 20:24:00.936268 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:00.936240    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj"
Nov 03 20:24:00.943475 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:00.943446    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k"]
Nov 03 20:24:00.963337 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:00.963306    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f"]
Nov 03 20:24:00.986644 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:00.984997    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8-plugins-config-volume\") pod \"6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8\" (UID: \"6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8\") "
Nov 03 20:24:00.986644 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:00.985081    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8-kube-api-access-hx647\" (UniqueName: \"kubernetes.io/projected/6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8-kube-api-access-hx647\") pod \"6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8\" (UID: \"6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8\") "
Nov 03 20:24:00.986644 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:00.985156    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3dbca1e-d679-4e9e-960d-9006b92b6d72-kube-api-access-2x4q8\" (UniqueName: \"kubernetes.io/projected/b3dbca1e-d679-4e9e-960d-9006b92b6d72-kube-api-access-2x4q8\") pod \"b3dbca1e-d679-4e9e-960d-9006b92b6d72\" (UID: \"b3dbca1e-d679-4e9e-960d-9006b92b6d72\") "
Nov 03 20:24:00.986644 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:00.985248    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3dbca1e-d679-4e9e-960d-9006b92b6d72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3dbca1e-d679-4e9e-960d-9006b92b6d72-plugins-config-volume\") pod \"b3dbca1e-d679-4e9e-960d-9006b92b6d72\" (UID: \"b3dbca1e-d679-4e9e-960d-9006b92b6d72\") "
Nov 03 20:24:00.986644 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:00.986479    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3dbca1e-d679-4e9e-960d-9006b92b6d72-plugins-config-volume" pod "b3dbca1e-d679-4e9e-960d-9006b92b6d72" (UID: "b3dbca1e-d679-4e9e-960d-9006b92b6d72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:24:00.986644 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:00.986638    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8-plugins-config-volume" pod "6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8" (UID: "6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:24:00.989391 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:00.989356    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3dbca1e-d679-4e9e-960d-9006b92b6d72-kube-api-access-2x4q8" pod "b3dbca1e-d679-4e9e-960d-9006b92b6d72" (UID: "b3dbca1e-d679-4e9e-960d-9006b92b6d72"). InnerVolumeSpecName "kube-api-access-2x4q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:24:00.991011 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:00.990988    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8-kube-api-access-hx647" pod "6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8" (UID: "6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8"). InnerVolumeSpecName "kube-api-access-hx647". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:24:01.007291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:01.007258    4421 generic.go:356] "Generic (PLEG): container finished" podID="b3dbca1e-d679-4e9e-960d-9006b92b6d72" containerID="27e621c3ee88618bf51812cd8b1c6758ee75a93d615adb282921e5193946a459" exitCode=0
Nov 03 20:24:01.007458 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:01.007354    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj" event={"ID":"b3dbca1e-d679-4e9e-960d-9006b92b6d72","Type":"ContainerDied","Data":"27e621c3ee88618bf51812cd8b1c6758ee75a93d615adb282921e5193946a459"}
Nov 03 20:24:01.007458 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:01.007397    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj" event={"ID":"b3dbca1e-d679-4e9e-960d-9006b92b6d72","Type":"ContainerDied","Data":"c422246fac0446bdc2b92b407befb7b920c4fbbaf462bf90532ef687e7db42a2"}
Nov 03 20:24:01.007458 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:01.007422    4421 scope.go:122] "RemoveContainer" containerID="27e621c3ee88618bf51812cd8b1c6758ee75a93d615adb282921e5193946a459"
Nov 03 20:24:01.007665 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:01.007570    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj"
Nov 03 20:24:01.016544 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:01.016515    4421 generic.go:356] "Generic (PLEG): container finished" podID="6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8" containerID="6dc54b7e81966bf70991beb151eca12e307b88340845ae711960b12a3c3fb230" exitCode=0
Nov 03 20:24:01.016716 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:01.016620    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv"
Nov 03 20:24:01.022739 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:01.022719    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6"
Nov 03 20:24:01.023617 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:01.023571    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6" event={"ID":"6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8","Type":"ContainerDied","Data":"6dc54b7e81966bf70991beb151eca12e307b88340845ae711960b12a3c3fb230"}
Nov 03 20:24:01.023922 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:01.023891    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6" event={"ID":"6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8","Type":"ContainerDied","Data":"a8465b6101c8806f35c8de76d3000dd2465f80e40f99c4a16cacc569286f4057"}
Nov 03 20:24:01.037217 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:01.037112    4421 scope.go:122] "RemoveContainer" containerID="27e621c3ee88618bf51812cd8b1c6758ee75a93d615adb282921e5193946a459"
Nov 03 20:24:01.037829 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:24:01.037800    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27e621c3ee88618bf51812cd8b1c6758ee75a93d615adb282921e5193946a459\": container with ID starting with 27e621c3ee88618bf51812cd8b1c6758ee75a93d615adb282921e5193946a459 not found: ID does not exist" containerID="27e621c3ee88618bf51812cd8b1c6758ee75a93d615adb282921e5193946a459"
Nov 03 20:24:01.037944 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:01.037841    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27e621c3ee88618bf51812cd8b1c6758ee75a93d615adb282921e5193946a459"} err="failed to get container status \"27e621c3ee88618bf51812cd8b1c6758ee75a93d615adb282921e5193946a459\": rpc error: code = NotFound desc = could not find container \"27e621c3ee88618bf51812cd8b1c6758ee75a93d615adb282921e5193946a459\": container with ID starting with 27e621c3ee88618bf51812cd8b1c6758ee75a93d615adb282921e5193946a459 not found: ID does not exist"
Nov 03 20:24:01.037944 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:01.037866    4421 scope.go:122] "RemoveContainer" containerID="6dc54b7e81966bf70991beb151eca12e307b88340845ae711960b12a3c3fb230"
Nov 03 20:24:01.053734 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:01.053711    4421 scope.go:122] "RemoveContainer" containerID="6dc54b7e81966bf70991beb151eca12e307b88340845ae711960b12a3c3fb230"
Nov 03 20:24:01.054106 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:24:01.054087    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dc54b7e81966bf70991beb151eca12e307b88340845ae711960b12a3c3fb230\": container with ID starting with 6dc54b7e81966bf70991beb151eca12e307b88340845ae711960b12a3c3fb230 not found: ID does not exist" containerID="6dc54b7e81966bf70991beb151eca12e307b88340845ae711960b12a3c3fb230"
Nov 03 20:24:01.054182 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:01.054114    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dc54b7e81966bf70991beb151eca12e307b88340845ae711960b12a3c3fb230"} err="failed to get container status \"6dc54b7e81966bf70991beb151eca12e307b88340845ae711960b12a3c3fb230\": rpc error: code = NotFound desc = could not find container \"6dc54b7e81966bf70991beb151eca12e307b88340845ae711960b12a3c3fb230\": container with ID starting with 6dc54b7e81966bf70991beb151eca12e307b88340845ae711960b12a3c3fb230 not found: ID does not exist"
Nov 03 20:24:01.056703 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:01.056266    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv"]
Nov 03 20:24:01.060159 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:01.060137    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c4f356-9889f64bb-swnfv"]
Nov 03 20:24:01.068558 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:01.068530    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj"]
Nov 03 20:24:01.076707 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:01.076687    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3f36ae-8479797959-ngkfj"]
Nov 03 20:24:01.086886 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:01.086864    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6"]
Nov 03 20:24:01.087641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:01.087619    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:01.087713 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:01.087645    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx647\" (UniqueName: \"kubernetes.io/projected/6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8-kube-api-access-hx647\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:01.087713 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:01.087655    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x4q8\" (UniqueName: \"kubernetes.io/projected/b3dbca1e-d679-4e9e-960d-9006b92b6d72-kube-api-access-2x4q8\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:01.087713 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:01.087664    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3dbca1e-d679-4e9e-960d-9006b92b6d72-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:01.105773 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:01.105680    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3f36ae-f89b78796-vtwt6"]
Nov 03 20:24:02.017570 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:02.017540    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8" path="/var/lib/kubelet/pods/6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8/volumes"
Nov 03 20:24:02.018041 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:02.018016    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3dbca1e-d679-4e9e-960d-9006b92b6d72" path="/var/lib/kubelet/pods/b3dbca1e-d679-4e9e-960d-9006b92b6d72/volumes"
Nov 03 20:24:02.018643 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:02.018620    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f479675b-17b7-432b-aef4-9cbc3bfb7359" path="/var/lib/kubelet/pods/f479675b-17b7-432b-aef4-9cbc3bfb7359/volumes"
Nov 03 20:24:02.531578 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:02.531542    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn" podUID="ae58dac5-e413-4ba5-8440-3d0b85bc9c7c" containerName="payload-processing" containerID="cri-o://0b143b94844ab8fd1108906eee599673589871636cafd850c6d6609a7acb406b" gracePeriod=25
Nov 03 20:24:02.824881 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:02.824861    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn"
Nov 03 20:24:02.859863 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:02.859819    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j" podUID="35b29971-9827-4d3b-800a-4d05f84d6b5d" containerName="payload-pre-processing" containerID="cri-o://b301df9685ef04a70fff1d0cfa391de06fd7bd656088233c193cedd0ff50bc11" gracePeriod=25
Nov 03 20:24:02.926660 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:02.926592    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae58dac5-e413-4ba5-8440-3d0b85bc9c7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae58dac5-e413-4ba5-8440-3d0b85bc9c7c-plugins-config-volume\") pod \"ae58dac5-e413-4ba5-8440-3d0b85bc9c7c\" (UID: \"ae58dac5-e413-4ba5-8440-3d0b85bc9c7c\") "
Nov 03 20:24:02.926803 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:02.926783    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae58dac5-e413-4ba5-8440-3d0b85bc9c7c-kube-api-access-22njh\" (UniqueName: \"kubernetes.io/projected/ae58dac5-e413-4ba5-8440-3d0b85bc9c7c-kube-api-access-22njh\") pod \"ae58dac5-e413-4ba5-8440-3d0b85bc9c7c\" (UID: \"ae58dac5-e413-4ba5-8440-3d0b85bc9c7c\") "
Nov 03 20:24:02.927035 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:02.927013    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae58dac5-e413-4ba5-8440-3d0b85bc9c7c-plugins-config-volume" pod "ae58dac5-e413-4ba5-8440-3d0b85bc9c7c" (UID: "ae58dac5-e413-4ba5-8440-3d0b85bc9c7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:24:02.927395 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:02.927374    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae58dac5-e413-4ba5-8440-3d0b85bc9c7c-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:02.928909 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:02.928886    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae58dac5-e413-4ba5-8440-3d0b85bc9c7c-kube-api-access-22njh" pod "ae58dac5-e413-4ba5-8440-3d0b85bc9c7c" (UID: "ae58dac5-e413-4ba5-8440-3d0b85bc9c7c"). InnerVolumeSpecName "kube-api-access-22njh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:24:03.036623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:03.032215    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22njh\" (UniqueName: \"kubernetes.io/projected/ae58dac5-e413-4ba5-8440-3d0b85bc9c7c-kube-api-access-22njh\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:03.064748 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:03.064670    4421 generic.go:356] "Generic (PLEG): container finished" podID="35b29971-9827-4d3b-800a-4d05f84d6b5d" containerID="b301df9685ef04a70fff1d0cfa391de06fd7bd656088233c193cedd0ff50bc11" exitCode=0
Nov 03 20:24:03.064908 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:03.064798    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j" event={"ID":"35b29971-9827-4d3b-800a-4d05f84d6b5d","Type":"ContainerDied","Data":"b301df9685ef04a70fff1d0cfa391de06fd7bd656088233c193cedd0ff50bc11"}
Nov 03 20:24:03.066487 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:03.066461    4421 generic.go:356] "Generic (PLEG): container finished" podID="ae58dac5-e413-4ba5-8440-3d0b85bc9c7c" containerID="0b143b94844ab8fd1108906eee599673589871636cafd850c6d6609a7acb406b" exitCode=0
Nov 03 20:24:03.066621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:03.066509    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn"
Nov 03 20:24:03.066621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:03.066499    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn" event={"ID":"ae58dac5-e413-4ba5-8440-3d0b85bc9c7c","Type":"ContainerDied","Data":"0b143b94844ab8fd1108906eee599673589871636cafd850c6d6609a7acb406b"}
Nov 03 20:24:03.066742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:03.066653    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn" event={"ID":"ae58dac5-e413-4ba5-8440-3d0b85bc9c7c","Type":"ContainerDied","Data":"0bea94c899df4c030f968d494a0330f468891f73caea2e2534d77a19537fec39"}
Nov 03 20:24:03.066742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:03.066722    4421 scope.go:122] "RemoveContainer" containerID="0b143b94844ab8fd1108906eee599673589871636cafd850c6d6609a7acb406b"
Nov 03 20:24:03.078515 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:03.078497    4421 scope.go:122] "RemoveContainer" containerID="0b143b94844ab8fd1108906eee599673589871636cafd850c6d6609a7acb406b"
Nov 03 20:24:03.078872 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:24:03.078854    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b143b94844ab8fd1108906eee599673589871636cafd850c6d6609a7acb406b\": container with ID starting with 0b143b94844ab8fd1108906eee599673589871636cafd850c6d6609a7acb406b not found: ID does not exist" containerID="0b143b94844ab8fd1108906eee599673589871636cafd850c6d6609a7acb406b"
Nov 03 20:24:03.078961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:03.078876    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b143b94844ab8fd1108906eee599673589871636cafd850c6d6609a7acb406b"} err="failed to get container status \"0b143b94844ab8fd1108906eee599673589871636cafd850c6d6609a7acb406b\": rpc error: code = NotFound desc = could not find container \"0b143b94844ab8fd1108906eee599673589871636cafd850c6d6609a7acb406b\": container with ID starting with 0b143b94844ab8fd1108906eee599673589871636cafd850c6d6609a7acb406b not found: ID does not exist"
Nov 03 20:24:03.095376 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:03.095351    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn"]
Nov 03 20:24:03.101153 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:03.101127    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c4f356-65984ccdd8-8sssn"]
Nov 03 20:24:03.197612 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:03.197564    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j"
Nov 03 20:24:03.234346 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:03.234318    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35b29971-9827-4d3b-800a-4d05f84d6b5d-kube-api-access-phq24\" (UniqueName: \"kubernetes.io/projected/35b29971-9827-4d3b-800a-4d05f84d6b5d-kube-api-access-phq24\") pod \"35b29971-9827-4d3b-800a-4d05f84d6b5d\" (UID: \"35b29971-9827-4d3b-800a-4d05f84d6b5d\") "
Nov 03 20:24:03.234496 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:03.234451    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35b29971-9827-4d3b-800a-4d05f84d6b5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35b29971-9827-4d3b-800a-4d05f84d6b5d-plugins-config-volume\") pod \"35b29971-9827-4d3b-800a-4d05f84d6b5d\" (UID: \"35b29971-9827-4d3b-800a-4d05f84d6b5d\") "
Nov 03 20:24:03.235032 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:03.235002    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b29971-9827-4d3b-800a-4d05f84d6b5d-plugins-config-volume" pod "35b29971-9827-4d3b-800a-4d05f84d6b5d" (UID: "35b29971-9827-4d3b-800a-4d05f84d6b5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:24:03.236943 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:03.236915    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b29971-9827-4d3b-800a-4d05f84d6b5d-kube-api-access-phq24" pod "35b29971-9827-4d3b-800a-4d05f84d6b5d" (UID: "35b29971-9827-4d3b-800a-4d05f84d6b5d"). InnerVolumeSpecName "kube-api-access-phq24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:24:03.318210 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:03.318126    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"]
Nov 03 20:24:03.318618 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:03.318423    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9" podUID="f8732447-9533-40b3-8da9-e8d6d946785e" containerName="istio-proxy" containerID="cri-o://0e9809d2925adf5d53a6e861bfc037814ecb8726578b55dde3128a83080d87aa" gracePeriod=30
Nov 03 20:24:03.336351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:03.336322    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phq24\" (UniqueName: \"kubernetes.io/projected/35b29971-9827-4d3b-800a-4d05f84d6b5d-kube-api-access-phq24\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:03.336351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:03.336353    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35b29971-9827-4d3b-800a-4d05f84d6b5d-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:04.017213 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:04.017176    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae58dac5-e413-4ba5-8440-3d0b85bc9c7c" path="/var/lib/kubelet/pods/ae58dac5-e413-4ba5-8440-3d0b85bc9c7c/volumes"
Nov 03 20:24:04.074897 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:04.074835    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j"
Nov 03 20:24:04.074897 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:04.074863    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j" event={"ID":"35b29971-9827-4d3b-800a-4d05f84d6b5d","Type":"ContainerDied","Data":"b006df236e084c9defcdd7b4a4fdcd90f5762975da2c548522e584a3bbed64a9"}
Nov 03 20:24:04.075308 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:04.074919    4421 scope.go:122] "RemoveContainer" containerID="b301df9685ef04a70fff1d0cfa391de06fd7bd656088233c193cedd0ff50bc11"
Nov 03 20:24:04.137147 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:04.137107    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j"]
Nov 03 20:24:04.140707 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:04.140683    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c4f356-54685dc96c-4v99j"]
Nov 03 20:24:05.946945 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:05.946590    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k" podUID="a53d3e21-8095-4e09-a80d-1b00732ea5b2" containerName="payload-processing" containerID="cri-o://03263ae82c60546392a7cbc7ba1950cfe54eacaf41c455e3e76057896a87a5f2" gracePeriod=25
Nov 03 20:24:05.967481 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:05.966428    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f" podUID="4103ec14-a57a-465d-b80b-0a76abe1870e" containerName="payload-pre-processing" containerID="cri-o://a5cd412b183e23a469b158ba7ec672aae4ecd85830959862f6f7aa59f44bd67a" gracePeriod=25
Nov 03 20:24:06.018189 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:06.018160    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35b29971-9827-4d3b-800a-4d05f84d6b5d" path="/var/lib/kubelet/pods/35b29971-9827-4d3b-800a-4d05f84d6b5d/volumes"
Nov 03 20:24:06.238928 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:06.238890    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"]
Nov 03 20:24:06.239212 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:06.239164    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq" podUID="436959f1-0deb-43cc-b8e0-ca75a6f0afbd" containerName="istio-proxy" containerID="cri-o://ec18ac3db0893853f2e1841f582006c0a793ff4c6d5927261fa1cc5faafe8f24" gracePeriod=30
Nov 03 20:24:06.316003 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:06.315510    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k"
Nov 03 20:24:06.443622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:06.437146    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a53d3e21-8095-4e09-a80d-1b00732ea5b2-kube-api-access-svdbl\" (UniqueName: \"kubernetes.io/projected/a53d3e21-8095-4e09-a80d-1b00732ea5b2-kube-api-access-svdbl\") pod \"a53d3e21-8095-4e09-a80d-1b00732ea5b2\" (UID: \"a53d3e21-8095-4e09-a80d-1b00732ea5b2\") "
Nov 03 20:24:06.443622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:06.437249    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a53d3e21-8095-4e09-a80d-1b00732ea5b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a53d3e21-8095-4e09-a80d-1b00732ea5b2-plugins-config-volume\") pod \"a53d3e21-8095-4e09-a80d-1b00732ea5b2\" (UID: \"a53d3e21-8095-4e09-a80d-1b00732ea5b2\") "
Nov 03 20:24:06.468669 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:06.465016    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a53d3e21-8095-4e09-a80d-1b00732ea5b2-plugins-config-volume" pod "a53d3e21-8095-4e09-a80d-1b00732ea5b2" (UID: "a53d3e21-8095-4e09-a80d-1b00732ea5b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:24:06.495625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:06.487452    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53d3e21-8095-4e09-a80d-1b00732ea5b2-kube-api-access-svdbl" pod "a53d3e21-8095-4e09-a80d-1b00732ea5b2" (UID: "a53d3e21-8095-4e09-a80d-1b00732ea5b2"). InnerVolumeSpecName "kube-api-access-svdbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:24:06.495625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:06.488969    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f"
Nov 03 20:24:06.552620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:06.539526    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4103ec14-a57a-465d-b80b-0a76abe1870e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4103ec14-a57a-465d-b80b-0a76abe1870e-plugins-config-volume\") pod \"4103ec14-a57a-465d-b80b-0a76abe1870e\" (UID: \"4103ec14-a57a-465d-b80b-0a76abe1870e\") "
Nov 03 20:24:06.552620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:06.540625    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4103ec14-a57a-465d-b80b-0a76abe1870e-plugins-config-volume" pod "4103ec14-a57a-465d-b80b-0a76abe1870e" (UID: "4103ec14-a57a-465d-b80b-0a76abe1870e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:24:06.552620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:06.550971    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4103ec14-a57a-465d-b80b-0a76abe1870e-kube-api-access-zjl6v\" (UniqueName: \"kubernetes.io/projected/4103ec14-a57a-465d-b80b-0a76abe1870e-kube-api-access-zjl6v\") pod \"4103ec14-a57a-465d-b80b-0a76abe1870e\" (UID: \"4103ec14-a57a-465d-b80b-0a76abe1870e\") "
Nov 03 20:24:06.552620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:06.551674    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4103ec14-a57a-465d-b80b-0a76abe1870e-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:06.552620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:06.551695    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svdbl\" (UniqueName: \"kubernetes.io/projected/a53d3e21-8095-4e09-a80d-1b00732ea5b2-kube-api-access-svdbl\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:06.552620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:06.551709    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a53d3e21-8095-4e09-a80d-1b00732ea5b2-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:06.561906 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:06.561812    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4103ec14-a57a-465d-b80b-0a76abe1870e-kube-api-access-zjl6v" pod "4103ec14-a57a-465d-b80b-0a76abe1870e" (UID: "4103ec14-a57a-465d-b80b-0a76abe1870e"). InnerVolumeSpecName "kube-api-access-zjl6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:24:06.653488 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:06.653450    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjl6v\" (UniqueName: \"kubernetes.io/projected/4103ec14-a57a-465d-b80b-0a76abe1870e-kube-api-access-zjl6v\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:07.102165 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:07.102132    4421 generic.go:356] "Generic (PLEG): container finished" podID="a53d3e21-8095-4e09-a80d-1b00732ea5b2" containerID="03263ae82c60546392a7cbc7ba1950cfe54eacaf41c455e3e76057896a87a5f2" exitCode=0
Nov 03 20:24:07.102631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:07.102183    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k"
Nov 03 20:24:07.102631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:07.102205    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k" event={"ID":"a53d3e21-8095-4e09-a80d-1b00732ea5b2","Type":"ContainerDied","Data":"03263ae82c60546392a7cbc7ba1950cfe54eacaf41c455e3e76057896a87a5f2"}
Nov 03 20:24:07.102631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:07.102256    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k" event={"ID":"a53d3e21-8095-4e09-a80d-1b00732ea5b2","Type":"ContainerDied","Data":"15ac737478e79a68456c359b1636ed5c4ff30e825655acf6b5127adc9eeda3d3"}
Nov 03 20:24:07.102631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:07.102280    4421 scope.go:122] "RemoveContainer" containerID="03263ae82c60546392a7cbc7ba1950cfe54eacaf41c455e3e76057896a87a5f2"
Nov 03 20:24:07.103789 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:07.103764    4421 generic.go:356] "Generic (PLEG): container finished" podID="4103ec14-a57a-465d-b80b-0a76abe1870e" containerID="a5cd412b183e23a469b158ba7ec672aae4ecd85830959862f6f7aa59f44bd67a" exitCode=0
Nov 03 20:24:07.103864 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:07.103803    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f"
Nov 03 20:24:07.103864 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:07.103831    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f" event={"ID":"4103ec14-a57a-465d-b80b-0a76abe1870e","Type":"ContainerDied","Data":"a5cd412b183e23a469b158ba7ec672aae4ecd85830959862f6f7aa59f44bd67a"}
Nov 03 20:24:07.103978 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:07.103872    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f" event={"ID":"4103ec14-a57a-465d-b80b-0a76abe1870e","Type":"ContainerDied","Data":"1d007e31f77f0b9545913e17c13718330e29f72c9f28e13460b06146b922e7a0"}
Nov 03 20:24:07.114368 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:07.114343    4421 scope.go:122] "RemoveContainer" containerID="03263ae82c60546392a7cbc7ba1950cfe54eacaf41c455e3e76057896a87a5f2"
Nov 03 20:24:07.114712 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:24:07.114691    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03263ae82c60546392a7cbc7ba1950cfe54eacaf41c455e3e76057896a87a5f2\": container with ID starting with 03263ae82c60546392a7cbc7ba1950cfe54eacaf41c455e3e76057896a87a5f2 not found: ID does not exist" containerID="03263ae82c60546392a7cbc7ba1950cfe54eacaf41c455e3e76057896a87a5f2"
Nov 03 20:24:07.114807 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:07.114717    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03263ae82c60546392a7cbc7ba1950cfe54eacaf41c455e3e76057896a87a5f2"} err="failed to get container status \"03263ae82c60546392a7cbc7ba1950cfe54eacaf41c455e3e76057896a87a5f2\": rpc error: code = NotFound desc = could not find container \"03263ae82c60546392a7cbc7ba1950cfe54eacaf41c455e3e76057896a87a5f2\": container with ID starting with 03263ae82c60546392a7cbc7ba1950cfe54eacaf41c455e3e76057896a87a5f2 not found: ID does not exist"
Nov 03 20:24:07.114807 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:07.114734    4421 scope.go:122] "RemoveContainer" containerID="a5cd412b183e23a469b158ba7ec672aae4ecd85830959862f6f7aa59f44bd67a"
Nov 03 20:24:07.124070 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:07.124053    4421 scope.go:122] "RemoveContainer" containerID="a5cd412b183e23a469b158ba7ec672aae4ecd85830959862f6f7aa59f44bd67a"
Nov 03 20:24:07.124329 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:24:07.124310    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5cd412b183e23a469b158ba7ec672aae4ecd85830959862f6f7aa59f44bd67a\": container with ID starting with a5cd412b183e23a469b158ba7ec672aae4ecd85830959862f6f7aa59f44bd67a not found: ID does not exist" containerID="a5cd412b183e23a469b158ba7ec672aae4ecd85830959862f6f7aa59f44bd67a"
Nov 03 20:24:07.124400 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:07.124339    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5cd412b183e23a469b158ba7ec672aae4ecd85830959862f6f7aa59f44bd67a"} err="failed to get container status \"a5cd412b183e23a469b158ba7ec672aae4ecd85830959862f6f7aa59f44bd67a\": rpc error: code = NotFound desc = could not find container \"a5cd412b183e23a469b158ba7ec672aae4ecd85830959862f6f7aa59f44bd67a\": container with ID starting with a5cd412b183e23a469b158ba7ec672aae4ecd85830959862f6f7aa59f44bd67a not found: ID does not exist"
Nov 03 20:24:07.130635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:07.130615    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k"]
Nov 03 20:24:07.135823 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:07.135805    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a596f7-5955dc7bc6-dqp4k"]
Nov 03 20:24:07.145861 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:07.145840    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f"]
Nov 03 20:24:07.152485 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:07.152465    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a596f7-6899469bf5-x8f4f"]
Nov 03 20:24:08.017133 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.017092    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4103ec14-a57a-465d-b80b-0a76abe1870e" path="/var/lib/kubelet/pods/4103ec14-a57a-465d-b80b-0a76abe1870e/volumes"
Nov 03 20:24:08.017692 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.017669    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a53d3e21-8095-4e09-a80d-1b00732ea5b2" path="/var/lib/kubelet/pods/a53d3e21-8095-4e09-a80d-1b00732ea5b2/volumes"
Nov 03 20:24:08.613337 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.613278    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:24:08.695035 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.695007    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-istio-data\") pod \"f8732447-9533-40b3-8da9-e8d6d946785e\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") "
Nov 03 20:24:08.695202 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.695129    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-workload-socket\") pod \"f8732447-9533-40b3-8da9-e8d6d946785e\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") "
Nov 03 20:24:08.695296 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.695282    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-credential-socket\") pod \"f8732447-9533-40b3-8da9-e8d6d946785e\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") "
Nov 03 20:24:08.695335 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.695309    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-istio-envoy\") pod \"f8732447-9533-40b3-8da9-e8d6d946785e\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") "
Nov 03 20:24:08.695387 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.695276    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-istio-data" pod "f8732447-9533-40b3-8da9-e8d6d946785e" (UID: "f8732447-9533-40b3-8da9-e8d6d946785e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:24:08.695387 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.695364    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-workload-socket" pod "f8732447-9533-40b3-8da9-e8d6d946785e" (UID: "f8732447-9533-40b3-8da9-e8d6d946785e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:24:08.695387 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.695382    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8732447-9533-40b3-8da9-e8d6d946785e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8732447-9533-40b3-8da9-e8d6d946785e-istio-podinfo\") pod \"f8732447-9533-40b3-8da9-e8d6d946785e\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") "
Nov 03 20:24:08.695518 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.695412    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8732447-9533-40b3-8da9-e8d6d946785e-istio-token\" (UniqueName: \"kubernetes.io/projected/f8732447-9533-40b3-8da9-e8d6d946785e-istio-token\") pod \"f8732447-9533-40b3-8da9-e8d6d946785e\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") "
Nov 03 20:24:08.695518 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.695454    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-workload-certs\") pod \"f8732447-9533-40b3-8da9-e8d6d946785e\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") "
Nov 03 20:24:08.695518 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.695494    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-credential-socket" pod "f8732447-9533-40b3-8da9-e8d6d946785e" (UID: "f8732447-9533-40b3-8da9-e8d6d946785e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:24:08.695518 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.695500    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8732447-9533-40b3-8da9-e8d6d946785e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8732447-9533-40b3-8da9-e8d6d946785e-istiod-ca-cert\") pod \"f8732447-9533-40b3-8da9-e8d6d946785e\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") "
Nov 03 20:24:08.695735 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.695584    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8732447-9533-40b3-8da9-e8d6d946785e-kube-api-access-w6dxl\" (UniqueName: \"kubernetes.io/projected/f8732447-9533-40b3-8da9-e8d6d946785e-kube-api-access-w6dxl\") pod \"f8732447-9533-40b3-8da9-e8d6d946785e\" (UID: \"f8732447-9533-40b3-8da9-e8d6d946785e\") "
Nov 03 20:24:08.695847 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.695821    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-workload-certs" pod "f8732447-9533-40b3-8da9-e8d6d946785e" (UID: "f8732447-9533-40b3-8da9-e8d6d946785e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:24:08.696359 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.696063    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8732447-9533-40b3-8da9-e8d6d946785e-istiod-ca-cert" pod "f8732447-9533-40b3-8da9-e8d6d946785e" (UID: "f8732447-9533-40b3-8da9-e8d6d946785e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:24:08.696522 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.696500    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:08.696614 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.696529    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:08.696614 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.696545    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8732447-9533-40b3-8da9-e8d6d946785e-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:08.696614 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.696561    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:08.696614 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.696575    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:08.698177 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.698150    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-istio-envoy" pod "f8732447-9533-40b3-8da9-e8d6d946785e" (UID: "f8732447-9533-40b3-8da9-e8d6d946785e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:24:08.698504 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.698481    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8732447-9533-40b3-8da9-e8d6d946785e-kube-api-access-w6dxl" pod "f8732447-9533-40b3-8da9-e8d6d946785e" (UID: "f8732447-9533-40b3-8da9-e8d6d946785e"). InnerVolumeSpecName "kube-api-access-w6dxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:24:08.698573 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.698552    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8732447-9533-40b3-8da9-e8d6d946785e-istio-token" pod "f8732447-9533-40b3-8da9-e8d6d946785e" (UID: "f8732447-9533-40b3-8da9-e8d6d946785e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:24:08.698799 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.698779    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8732447-9533-40b3-8da9-e8d6d946785e-istio-podinfo" pod "f8732447-9533-40b3-8da9-e8d6d946785e" (UID: "f8732447-9533-40b3-8da9-e8d6d946785e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:24:08.797699 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.797671    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8732447-9533-40b3-8da9-e8d6d946785e-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:08.797699 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.797696    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8732447-9533-40b3-8da9-e8d6d946785e-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:08.797878 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.797705    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6dxl\" (UniqueName: \"kubernetes.io/projected/f8732447-9533-40b3-8da9-e8d6d946785e-kube-api-access-w6dxl\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:08.797878 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:08.797715    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8732447-9533-40b3-8da9-e8d6d946785e-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:09.093122 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.093076    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8"]
Nov 03 20:24:09.094023 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.093349    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8" podUID="25115ccd-00e2-48d2-8202-654e4920800f" containerName="maas-api" containerID="cri-o://6d9be67eee6cce135a4fca1fb2842903b3b418de822f6ba20695c77db088facd" gracePeriod=30
Nov 03 20:24:09.126801 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.126772    4421 generic.go:356] "Generic (PLEG): container finished" podID="f8732447-9533-40b3-8da9-e8d6d946785e" containerID="0e9809d2925adf5d53a6e861bfc037814ecb8726578b55dde3128a83080d87aa" exitCode=0
Nov 03 20:24:09.126953 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.126881    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"
Nov 03 20:24:09.127014 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.126948    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9" event={"ID":"f8732447-9533-40b3-8da9-e8d6d946785e","Type":"ContainerDied","Data":"0e9809d2925adf5d53a6e861bfc037814ecb8726578b55dde3128a83080d87aa"}
Nov 03 20:24:09.127014 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.126991    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9" event={"ID":"f8732447-9533-40b3-8da9-e8d6d946785e","Type":"ContainerDied","Data":"f4c583c9d80f34875fa28c297d3e0bd2a28e6df0cfb6624c22ee62762aeb9b52"}
Nov 03 20:24:09.127135 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.127016    4421 scope.go:122] "RemoveContainer" containerID="0e9809d2925adf5d53a6e861bfc037814ecb8726578b55dde3128a83080d87aa"
Nov 03 20:24:09.157464 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.157440    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"]
Nov 03 20:24:09.160854 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.160832    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-3f36ae-openshift-default-657644fd77-v4fw9"]
Nov 03 20:24:09.207685 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.207651    4421 scope.go:122] "RemoveContainer" containerID="73120e7fc615b4e82e5509496d1f0d1f8614ff615bd7e4efc2a949681dbc8f11"
Nov 03 20:24:09.225787 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.225761    4421 scope.go:122] "RemoveContainer" containerID="0e9809d2925adf5d53a6e861bfc037814ecb8726578b55dde3128a83080d87aa"
Nov 03 20:24:09.226192 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:24:09.226168    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e9809d2925adf5d53a6e861bfc037814ecb8726578b55dde3128a83080d87aa\": container with ID starting with 0e9809d2925adf5d53a6e861bfc037814ecb8726578b55dde3128a83080d87aa not found: ID does not exist" containerID="0e9809d2925adf5d53a6e861bfc037814ecb8726578b55dde3128a83080d87aa"
Nov 03 20:24:09.226289 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.226209    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e9809d2925adf5d53a6e861bfc037814ecb8726578b55dde3128a83080d87aa"} err="failed to get container status \"0e9809d2925adf5d53a6e861bfc037814ecb8726578b55dde3128a83080d87aa\": rpc error: code = NotFound desc = could not find container \"0e9809d2925adf5d53a6e861bfc037814ecb8726578b55dde3128a83080d87aa\": container with ID starting with 0e9809d2925adf5d53a6e861bfc037814ecb8726578b55dde3128a83080d87aa not found: ID does not exist"
Nov 03 20:24:09.226289 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.226236    4421 scope.go:122] "RemoveContainer" containerID="73120e7fc615b4e82e5509496d1f0d1f8614ff615bd7e4efc2a949681dbc8f11"
Nov 03 20:24:09.226588 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:24:09.226569    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73120e7fc615b4e82e5509496d1f0d1f8614ff615bd7e4efc2a949681dbc8f11\": container with ID starting with 73120e7fc615b4e82e5509496d1f0d1f8614ff615bd7e4efc2a949681dbc8f11 not found: ID does not exist" containerID="73120e7fc615b4e82e5509496d1f0d1f8614ff615bd7e4efc2a949681dbc8f11"
Nov 03 20:24:09.226680 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.226632    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73120e7fc615b4e82e5509496d1f0d1f8614ff615bd7e4efc2a949681dbc8f11"} err="failed to get container status \"73120e7fc615b4e82e5509496d1f0d1f8614ff615bd7e4efc2a949681dbc8f11\": rpc error: code = NotFound desc = could not find container \"73120e7fc615b4e82e5509496d1f0d1f8614ff615bd7e4efc2a949681dbc8f11\": container with ID starting with 73120e7fc615b4e82e5509496d1f0d1f8614ff615bd7e4efc2a949681dbc8f11 not found: ID does not exist"
Nov 03 20:24:09.625278 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.615872    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8"
Nov 03 20:24:09.689968 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.689933    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj"]
Nov 03 20:24:09.748711 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.748681    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25115ccd-00e2-48d2-8202-654e4920800f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25115ccd-00e2-48d2-8202-654e4920800f-maas-api-tls\") pod \"25115ccd-00e2-48d2-8202-654e4920800f\" (UID: \"25115ccd-00e2-48d2-8202-654e4920800f\") "
Nov 03 20:24:09.748869 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.748799    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25115ccd-00e2-48d2-8202-654e4920800f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25115ccd-00e2-48d2-8202-654e4920800f-openshift-service-ca\") pod \"25115ccd-00e2-48d2-8202-654e4920800f\" (UID: \"25115ccd-00e2-48d2-8202-654e4920800f\") "
Nov 03 20:24:09.748869 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.748846    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25115ccd-00e2-48d2-8202-654e4920800f-kube-api-access-j776s\" (UniqueName: \"kubernetes.io/projected/25115ccd-00e2-48d2-8202-654e4920800f-kube-api-access-j776s\") pod \"25115ccd-00e2-48d2-8202-654e4920800f\" (UID: \"25115ccd-00e2-48d2-8202-654e4920800f\") "
Nov 03 20:24:09.748965 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.748879    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25115ccd-00e2-48d2-8202-654e4920800f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/25115ccd-00e2-48d2-8202-654e4920800f-metrics-certs\") pod \"25115ccd-00e2-48d2-8202-654e4920800f\" (UID: \"25115ccd-00e2-48d2-8202-654e4920800f\") "
Nov 03 20:24:09.749218 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.749193    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25115ccd-00e2-48d2-8202-654e4920800f-openshift-service-ca" pod "25115ccd-00e2-48d2-8202-654e4920800f" (UID: "25115ccd-00e2-48d2-8202-654e4920800f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:24:09.749779 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.749737    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25115ccd-00e2-48d2-8202-654e4920800f-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:09.751375 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.751351    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25115ccd-00e2-48d2-8202-654e4920800f-metrics-certs" pod "25115ccd-00e2-48d2-8202-654e4920800f" (UID: "25115ccd-00e2-48d2-8202-654e4920800f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:24:09.751540 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.751520    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25115ccd-00e2-48d2-8202-654e4920800f-maas-api-tls" pod "25115ccd-00e2-48d2-8202-654e4920800f" (UID: "25115ccd-00e2-48d2-8202-654e4920800f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:24:09.751655 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.751593    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25115ccd-00e2-48d2-8202-654e4920800f-kube-api-access-j776s" pod "25115ccd-00e2-48d2-8202-654e4920800f" (UID: "25115ccd-00e2-48d2-8202-654e4920800f"). InnerVolumeSpecName "kube-api-access-j776s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:24:09.851181 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.851153    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25115ccd-00e2-48d2-8202-654e4920800f-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:09.851181 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.851181    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j776s\" (UniqueName: \"kubernetes.io/projected/25115ccd-00e2-48d2-8202-654e4920800f-kube-api-access-j776s\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:09.851369 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:09.851194    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25115ccd-00e2-48d2-8202-654e4920800f-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:10.073633 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:10.071443    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8732447-9533-40b3-8da9-e8d6d946785e" path="/var/lib/kubelet/pods/f8732447-9533-40b3-8da9-e8d6d946785e/volumes"
Nov 03 20:24:10.160710 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:10.160675    4421 generic.go:356] "Generic (PLEG): container finished" podID="25115ccd-00e2-48d2-8202-654e4920800f" containerID="6d9be67eee6cce135a4fca1fb2842903b3b418de822f6ba20695c77db088facd" exitCode=0
Nov 03 20:24:10.160843 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:10.160797    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8" event={"ID":"25115ccd-00e2-48d2-8202-654e4920800f","Type":"ContainerDied","Data":"6d9be67eee6cce135a4fca1fb2842903b3b418de822f6ba20695c77db088facd"}
Nov 03 20:24:10.161307 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:10.160851    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8" event={"ID":"25115ccd-00e2-48d2-8202-654e4920800f","Type":"ContainerDied","Data":"e93dd075405392ffdf52d1b9441b73832866fd5db4ff893a59830fc3eb9b1c41"}
Nov 03 20:24:10.161307 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:10.160880    4421 scope.go:122] "RemoveContainer" containerID="6d9be67eee6cce135a4fca1fb2842903b3b418de822f6ba20695c77db088facd"
Nov 03 20:24:10.161307 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:10.160991    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8"
Nov 03 20:24:10.176473 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:10.176449    4421 scope.go:122] "RemoveContainer" containerID="6d9be67eee6cce135a4fca1fb2842903b3b418de822f6ba20695c77db088facd"
Nov 03 20:24:10.176804 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:24:10.176784    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d9be67eee6cce135a4fca1fb2842903b3b418de822f6ba20695c77db088facd\": container with ID starting with 6d9be67eee6cce135a4fca1fb2842903b3b418de822f6ba20695c77db088facd not found: ID does not exist" containerID="6d9be67eee6cce135a4fca1fb2842903b3b418de822f6ba20695c77db088facd"
Nov 03 20:24:10.176876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:10.176813    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d9be67eee6cce135a4fca1fb2842903b3b418de822f6ba20695c77db088facd"} err="failed to get container status \"6d9be67eee6cce135a4fca1fb2842903b3b418de822f6ba20695c77db088facd\": rpc error: code = NotFound desc = could not find container \"6d9be67eee6cce135a4fca1fb2842903b3b418de822f6ba20695c77db088facd\": container with ID starting with 6d9be67eee6cce135a4fca1fb2842903b3b418de822f6ba20695c77db088facd not found: ID does not exist"
Nov 03 20:24:10.181925 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:10.181896    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8"]
Nov 03 20:24:10.186510 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:10.186486    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a596f7-6bbb4d68c6-2bdz8"]
Nov 03 20:24:11.031296 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.027648    4421 scope.go:122] "RemoveContainer" containerID="5dd0ee0556afec28673665ff12fbc9a052d911726b87e6abfd158dc5aed0a8da"
Nov 03 20:24:11.589174 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.589147    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:24:11.698867 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.698842    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-token\" (UniqueName: \"kubernetes.io/projected/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-token\") pod \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") "
Nov 03 20:24:11.699032 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.698886    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-podinfo\") pod \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") "
Nov 03 20:24:11.699032 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.698924    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-envoy\") pod \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") "
Nov 03 20:24:11.699032 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.698958    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-data\") pod \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") "
Nov 03 20:24:11.699032 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.698990    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-workload-socket\") pod \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") "
Nov 03 20:24:11.699261 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.699098    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-credential-socket\") pod \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") "
Nov 03 20:24:11.699261 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.699165    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istiod-ca-cert\") pod \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") "
Nov 03 20:24:11.699261 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.699247    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-kube-api-access-wg5vk\" (UniqueName: \"kubernetes.io/projected/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-kube-api-access-wg5vk\") pod \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") "
Nov 03 20:24:11.699466 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.699289    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-workload-certs\") pod \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\" (UID: \"436959f1-0deb-43cc-b8e0-ca75a6f0afbd\") "
Nov 03 20:24:11.699466 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.699291    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-workload-socket" pod "436959f1-0deb-43cc-b8e0-ca75a6f0afbd" (UID: "436959f1-0deb-43cc-b8e0-ca75a6f0afbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:24:11.699584 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.699464    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-credential-socket" pod "436959f1-0deb-43cc-b8e0-ca75a6f0afbd" (UID: "436959f1-0deb-43cc-b8e0-ca75a6f0afbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:24:11.699854 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.699692    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-workload-certs" pod "436959f1-0deb-43cc-b8e0-ca75a6f0afbd" (UID: "436959f1-0deb-43cc-b8e0-ca75a6f0afbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:24:11.699854 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.699783    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-data" pod "436959f1-0deb-43cc-b8e0-ca75a6f0afbd" (UID: "436959f1-0deb-43cc-b8e0-ca75a6f0afbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:24:11.699854 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.699824    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istiod-ca-cert" pod "436959f1-0deb-43cc-b8e0-ca75a6f0afbd" (UID: "436959f1-0deb-43cc-b8e0-ca75a6f0afbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:24:11.700188 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.700159    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:11.700241 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.700188    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:11.700241 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.700205    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:11.700241 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.700220    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:11.700241 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.700235    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:11.701766 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.701516    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-token" pod "436959f1-0deb-43cc-b8e0-ca75a6f0afbd" (UID: "436959f1-0deb-43cc-b8e0-ca75a6f0afbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:24:11.701930 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.701896    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-podinfo" pod "436959f1-0deb-43cc-b8e0-ca75a6f0afbd" (UID: "436959f1-0deb-43cc-b8e0-ca75a6f0afbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:24:11.702222 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.702203    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-kube-api-access-wg5vk" pod "436959f1-0deb-43cc-b8e0-ca75a6f0afbd" (UID: "436959f1-0deb-43cc-b8e0-ca75a6f0afbd"). InnerVolumeSpecName "kube-api-access-wg5vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:24:11.702935 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.702912    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-envoy" pod "436959f1-0deb-43cc-b8e0-ca75a6f0afbd" (UID: "436959f1-0deb-43cc-b8e0-ca75a6f0afbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:24:11.802186 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.802148    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg5vk\" (UniqueName: \"kubernetes.io/projected/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-kube-api-access-wg5vk\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:11.802186 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.802181    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:11.802186 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.802193    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:11.802430 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:11.802207    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/436959f1-0deb-43cc-b8e0-ca75a6f0afbd-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:12.017220 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:12.017144    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25115ccd-00e2-48d2-8202-654e4920800f" path="/var/lib/kubelet/pods/25115ccd-00e2-48d2-8202-654e4920800f/volumes"
Nov 03 20:24:12.235676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:12.235637    4421 generic.go:356] "Generic (PLEG): container finished" podID="436959f1-0deb-43cc-b8e0-ca75a6f0afbd" containerID="ec18ac3db0893853f2e1841f582006c0a793ff4c6d5927261fa1cc5faafe8f24" exitCode=0
Nov 03 20:24:12.235676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:12.235664    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"
Nov 03 20:24:12.236206 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:12.235695    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq" event={"ID":"436959f1-0deb-43cc-b8e0-ca75a6f0afbd","Type":"ContainerDied","Data":"ec18ac3db0893853f2e1841f582006c0a793ff4c6d5927261fa1cc5faafe8f24"}
Nov 03 20:24:12.236206 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:12.235746    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq" event={"ID":"436959f1-0deb-43cc-b8e0-ca75a6f0afbd","Type":"ContainerDied","Data":"049931a7939314cd179d9e5e049c3a23400440eb1d4b9e8d561c537f95f9cd63"}
Nov 03 20:24:12.236206 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:12.235776    4421 scope.go:122] "RemoveContainer" containerID="ec18ac3db0893853f2e1841f582006c0a793ff4c6d5927261fa1cc5faafe8f24"
Nov 03 20:24:12.273113 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:12.273093    4421 scope.go:122] "RemoveContainer" containerID="2a9a4ea1e1b460a1696e95d47270d6869143cb09f49d12564b190da7bd639219"
Nov 03 20:24:12.280003 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:12.279964    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"]
Nov 03 20:24:12.283186 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:12.283164    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-c4f356-openshift-default-845ddfdbdf-dhlxq"]
Nov 03 20:24:12.284544 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:12.284528    4421 scope.go:122] "RemoveContainer" containerID="ec18ac3db0893853f2e1841f582006c0a793ff4c6d5927261fa1cc5faafe8f24"
Nov 03 20:24:12.284890 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:24:12.284873    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec18ac3db0893853f2e1841f582006c0a793ff4c6d5927261fa1cc5faafe8f24\": container with ID starting with ec18ac3db0893853f2e1841f582006c0a793ff4c6d5927261fa1cc5faafe8f24 not found: ID does not exist" containerID="ec18ac3db0893853f2e1841f582006c0a793ff4c6d5927261fa1cc5faafe8f24"
Nov 03 20:24:12.284952 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:12.284897    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec18ac3db0893853f2e1841f582006c0a793ff4c6d5927261fa1cc5faafe8f24"} err="failed to get container status \"ec18ac3db0893853f2e1841f582006c0a793ff4c6d5927261fa1cc5faafe8f24\": rpc error: code = NotFound desc = could not find container \"ec18ac3db0893853f2e1841f582006c0a793ff4c6d5927261fa1cc5faafe8f24\": container with ID starting with ec18ac3db0893853f2e1841f582006c0a793ff4c6d5927261fa1cc5faafe8f24 not found: ID does not exist"
Nov 03 20:24:12.284952 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:12.284914    4421 scope.go:122] "RemoveContainer" containerID="2a9a4ea1e1b460a1696e95d47270d6869143cb09f49d12564b190da7bd639219"
Nov 03 20:24:12.285206 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:24:12.285188    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a9a4ea1e1b460a1696e95d47270d6869143cb09f49d12564b190da7bd639219\": container with ID starting with 2a9a4ea1e1b460a1696e95d47270d6869143cb09f49d12564b190da7bd639219 not found: ID does not exist" containerID="2a9a4ea1e1b460a1696e95d47270d6869143cb09f49d12564b190da7bd639219"
Nov 03 20:24:12.285261 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:12.285216    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a9a4ea1e1b460a1696e95d47270d6869143cb09f49d12564b190da7bd639219"} err="failed to get container status \"2a9a4ea1e1b460a1696e95d47270d6869143cb09f49d12564b190da7bd639219\": rpc error: code = NotFound desc = could not find container \"2a9a4ea1e1b460a1696e95d47270d6869143cb09f49d12564b190da7bd639219\": container with ID starting with 2a9a4ea1e1b460a1696e95d47270d6869143cb09f49d12564b190da7bd639219 not found: ID does not exist"
Nov 03 20:24:13.279973 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.279928    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"]
Nov 03 20:24:13.280885 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.280863    4421 cpu_manager.go:414] "containerMap: removing container" podUID="4103ec14-a57a-465d-b80b-0a76abe1870e" containerName="payload-pre-processing"
Nov 03 20:24:13.280979 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.280888    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4103ec14-a57a-465d-b80b-0a76abe1870e" containerName="payload-pre-processing"
Nov 03 20:24:13.280979 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.280923    4421 cpu_manager.go:414] "containerMap: removing container" podUID="c263b0e4-5dd7-4a46-a527-f461b779e42f" containerName="istio-proxy"
Nov 03 20:24:13.280979 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.280933    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c263b0e4-5dd7-4a46-a527-f461b779e42f" containerName="istio-proxy"
Nov 03 20:24:13.280979 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.280945    4421 cpu_manager.go:414] "containerMap: removing container" podUID="f479675b-17b7-432b-aef4-9cbc3bfb7359" containerName="maas-api"
Nov 03 20:24:13.281218 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281004    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f479675b-17b7-432b-aef4-9cbc3bfb7359" containerName="maas-api"
Nov 03 20:24:13.281218 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281016    4421 cpu_manager.go:414] "containerMap: removing container" podUID="f8732447-9533-40b3-8da9-e8d6d946785e" containerName="istio-proxy"
Nov 03 20:24:13.281218 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281024    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8732447-9533-40b3-8da9-e8d6d946785e" containerName="istio-proxy"
Nov 03 20:24:13.281218 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281046    4421 cpu_manager.go:414] "containerMap: removing container" podUID="35b29971-9827-4d3b-800a-4d05f84d6b5d" containerName="payload-pre-processing"
Nov 03 20:24:13.281218 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281053    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35b29971-9827-4d3b-800a-4d05f84d6b5d" containerName="payload-pre-processing"
Nov 03 20:24:13.281218 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281101    4421 cpu_manager.go:414] "containerMap: removing container" podUID="436959f1-0deb-43cc-b8e0-ca75a6f0afbd" containerName="istio-proxy"
Nov 03 20:24:13.281218 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281108    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="436959f1-0deb-43cc-b8e0-ca75a6f0afbd" containerName="istio-proxy"
Nov 03 20:24:13.281218 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281120    4421 cpu_manager.go:414] "containerMap: removing container" podUID="a53d3e21-8095-4e09-a80d-1b00732ea5b2" containerName="payload-processing"
Nov 03 20:24:13.281218 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281127    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53d3e21-8095-4e09-a80d-1b00732ea5b2" containerName="payload-processing"
Nov 03 20:24:13.281218 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281142    4421 cpu_manager.go:414] "containerMap: removing container" podUID="436959f1-0deb-43cc-b8e0-ca75a6f0afbd" containerName="istio-proxy"
Nov 03 20:24:13.281218 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281148    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="436959f1-0deb-43cc-b8e0-ca75a6f0afbd" containerName="istio-proxy"
Nov 03 20:24:13.281218 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281158    4421 cpu_manager.go:414] "containerMap: removing container" podUID="ae58dac5-e413-4ba5-8440-3d0b85bc9c7c" containerName="payload-processing"
Nov 03 20:24:13.281218 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281164    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae58dac5-e413-4ba5-8440-3d0b85bc9c7c" containerName="payload-processing"
Nov 03 20:24:13.281218 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281175    4421 cpu_manager.go:414] "containerMap: removing container" podUID="6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8" containerName="payload-processing"
Nov 03 20:24:13.281218 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281181    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8" containerName="payload-processing"
Nov 03 20:24:13.281218 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281192    4421 cpu_manager.go:414] "containerMap: removing container" podUID="25115ccd-00e2-48d2-8202-654e4920800f" containerName="maas-api"
Nov 03 20:24:13.281218 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281197    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25115ccd-00e2-48d2-8202-654e4920800f" containerName="maas-api"
Nov 03 20:24:13.281218 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281207    4421 cpu_manager.go:414] "containerMap: removing container" podUID="c263b0e4-5dd7-4a46-a527-f461b779e42f" containerName="istio-proxy"
Nov 03 20:24:13.281218 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281214    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c263b0e4-5dd7-4a46-a527-f461b779e42f" containerName="istio-proxy"
Nov 03 20:24:13.281218 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281228    4421 cpu_manager.go:414] "containerMap: removing container" podUID="b3dbca1e-d679-4e9e-960d-9006b92b6d72" containerName="payload-pre-processing"
Nov 03 20:24:13.282254 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281235    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3dbca1e-d679-4e9e-960d-9006b92b6d72" containerName="payload-pre-processing"
Nov 03 20:24:13.282254 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281255    4421 cpu_manager.go:414] "containerMap: removing container" podUID="6b34f6ef-a056-4dbb-9e6f-d61c78d770b6" containerName="istio-proxy"
Nov 03 20:24:13.282254 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281262    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b34f6ef-a056-4dbb-9e6f-d61c78d770b6" containerName="istio-proxy"
Nov 03 20:24:13.282254 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281418    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53d3e21-8095-4e09-a80d-1b00732ea5b2" containerName="payload-processing"
Nov 03 20:24:13.282254 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281431    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="436959f1-0deb-43cc-b8e0-ca75a6f0afbd" containerName="istio-proxy"
Nov 03 20:24:13.282254 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281444    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae58dac5-e413-4ba5-8440-3d0b85bc9c7c" containerName="payload-processing"
Nov 03 20:24:13.282254 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281452    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dfee168-d0d0-4f82-bcc3-3a2bf0b0b1d8" containerName="payload-processing"
Nov 03 20:24:13.282254 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281462    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="25115ccd-00e2-48d2-8202-654e4920800f" containerName="maas-api"
Nov 03 20:24:13.282254 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281473    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c263b0e4-5dd7-4a46-a527-f461b779e42f" containerName="istio-proxy"
Nov 03 20:24:13.282254 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281483    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3dbca1e-d679-4e9e-960d-9006b92b6d72" containerName="payload-pre-processing"
Nov 03 20:24:13.282254 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281491    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b34f6ef-a056-4dbb-9e6f-d61c78d770b6" containerName="istio-proxy"
Nov 03 20:24:13.282254 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281502    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8732447-9533-40b3-8da9-e8d6d946785e" containerName="istio-proxy"
Nov 03 20:24:13.282254 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281514    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="4103ec14-a57a-465d-b80b-0a76abe1870e" containerName="payload-pre-processing"
Nov 03 20:24:13.282254 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281523    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f479675b-17b7-432b-aef4-9cbc3bfb7359" containerName="maas-api"
Nov 03 20:24:13.282254 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.281532    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="35b29971-9827-4d3b-800a-4d05f84d6b5d" containerName="payload-pre-processing"
Nov 03 20:24:13.288075 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.288044    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.293804 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.293776    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-59bc3b-openshift-default-dockercfg-xd8nj\""
Nov 03 20:24:13.302319 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.302294    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"]
Nov 03 20:24:13.508623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.499992    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-credential-socket\") pod \"e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") " pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.508623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.500157    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-data\") pod \"e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") " pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.508623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.500237    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-workload-certs\") pod \"e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") " pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.508623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.500334    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-envoy\") pod \"e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") " pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.508623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.500415    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-token\") pod \"e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") " pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.508623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.500440    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-workload-socket\") pod \"e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") " pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.508623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.504722    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istiod-ca-cert\") pod \"e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") " pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.508623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.504828    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvl55\" (UniqueName: \"kubernetes.io/projected/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-kube-api-access-wvl55\") pod \"e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") " pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.508623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.504956    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-podinfo\") pod \"e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") " pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.637332 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.636740    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-credential-socket\") pod \"e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") " pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.637332 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.636814    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-data\") pod \"e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") " pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.637332 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.636843    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-workload-certs\") pod \"e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") " pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.637332 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.636898    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-envoy\") pod \"e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") " pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.637332 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.636952    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-token\") pod \"e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") " pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.637332 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.636975    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-workload-socket\") pod \"e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") " pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.637332 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.637039    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istiod-ca-cert\") pod \"e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") " pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.637332 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.637093    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvl55\" (UniqueName: \"kubernetes.io/projected/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-kube-api-access-wvl55\") pod \"e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") " pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.637332 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.637165    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-podinfo\") pod \"e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") " pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.671810 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.656906    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-data\") pod \"e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") " pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.671810 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.657176    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-workload-socket\") pod \"e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") " pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.671810 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.658109    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istiod-ca-cert\") pod \"e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") " pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.671810 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.658352    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-workload-certs\") pod \"e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") " pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.672892 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.672854    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-podinfo\") pod \"e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") " pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.673933 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.673909    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-credential-socket\") pod \"e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") " pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.678709 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.675343    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-envoy\") pod \"e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") " pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.680211 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.679444    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-token\") pod \"e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") " pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.681096 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.680886    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvl55\" (UniqueName: \"kubernetes.io/projected/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-kube-api-access-wvl55\") pod \"e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") " pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:13.925609 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:13.925514    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:14.024656 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:14.021176    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="436959f1-0deb-43cc-b8e0-ca75a6f0afbd" path="/var/lib/kubelet/pods/436959f1-0deb-43cc-b8e0-ca75a6f0afbd/volumes"
Nov 03 20:24:14.271759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:14.270734    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"]
Nov 03 20:24:14.282711 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:14.282655    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:24:14.283352 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:14.283312    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:24:14.283502 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:14.283486    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:24:14.296190 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:14.296154    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5" event={"ID":"8e6a0a0b-c9ca-4182-8962-fd4534a2844b","Type":"ContainerStarted","Data":"900f94ce3388fb55026f045963e51799c6eebfd8f86bfdd922c2fac9eb378b86"}
Nov 03 20:24:14.691042 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:14.690992    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj" podUID="a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5" containerName="payload-processing" containerID="cri-o://6d6ab5611ab9b5e9766edbe1bc297fea9b7374ed8823260c6a97273b5ddce19f" gracePeriod=25
Nov 03 20:24:15.110450 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:15.110427    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj"
Nov 03 20:24:15.194525 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:15.194493    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5-plugins-config-volume\") pod \"a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5\" (UID: \"a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5\") "
Nov 03 20:24:15.194777 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:15.194611    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5-kube-api-access-xlf6j\" (UniqueName: \"kubernetes.io/projected/a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5-kube-api-access-xlf6j\") pod \"a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5\" (UID: \"a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5\") "
Nov 03 20:24:15.195034 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:15.195003    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5-plugins-config-volume" pod "a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5" (UID: "a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:24:15.211091 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:15.197459    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5-kube-api-access-xlf6j" pod "a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5" (UID: "a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5"). InnerVolumeSpecName "kube-api-access-xlf6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:24:15.296554 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:15.296513    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlf6j\" (UniqueName: \"kubernetes.io/projected/a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5-kube-api-access-xlf6j\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:15.296554 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:15.296548    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:15.306579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:15.306547    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5" event={"ID":"8e6a0a0b-c9ca-4182-8962-fd4534a2844b","Type":"ContainerStarted","Data":"e563aea5ed0451dc85ae7b158bd943a4689594239571b31a5583cf922ea4f331"}
Nov 03 20:24:15.309183 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:15.308675    4421 generic.go:356] "Generic (PLEG): container finished" podID="a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5" containerID="6d6ab5611ab9b5e9766edbe1bc297fea9b7374ed8823260c6a97273b5ddce19f" exitCode=0
Nov 03 20:24:15.309183 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:15.308842    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj" event={"ID":"a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5","Type":"ContainerDied","Data":"6d6ab5611ab9b5e9766edbe1bc297fea9b7374ed8823260c6a97273b5ddce19f"}
Nov 03 20:24:15.309183 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:15.308878    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj" event={"ID":"a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5","Type":"ContainerDied","Data":"3b31d8c683102130cea0f06ca90a816fe79b7aba3caef32e4a555406e8c318e0"}
Nov 03 20:24:15.309183 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:15.308900    4421 scope.go:122] "RemoveContainer" containerID="6d6ab5611ab9b5e9766edbe1bc297fea9b7374ed8823260c6a97273b5ddce19f"
Nov 03 20:24:15.309183 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:15.308999    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj"
Nov 03 20:24:15.327309 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:15.327253    4421 scope.go:122] "RemoveContainer" containerID="6d6ab5611ab9b5e9766edbe1bc297fea9b7374ed8823260c6a97273b5ddce19f"
Nov 03 20:24:15.327638 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:24:15.327571    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d6ab5611ab9b5e9766edbe1bc297fea9b7374ed8823260c6a97273b5ddce19f\": container with ID starting with 6d6ab5611ab9b5e9766edbe1bc297fea9b7374ed8823260c6a97273b5ddce19f not found: ID does not exist" containerID="6d6ab5611ab9b5e9766edbe1bc297fea9b7374ed8823260c6a97273b5ddce19f"
Nov 03 20:24:15.327638 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:15.327628    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d6ab5611ab9b5e9766edbe1bc297fea9b7374ed8823260c6a97273b5ddce19f"} err="failed to get container status \"6d6ab5611ab9b5e9766edbe1bc297fea9b7374ed8823260c6a97273b5ddce19f\": rpc error: code = NotFound desc = could not find container \"6d6ab5611ab9b5e9766edbe1bc297fea9b7374ed8823260c6a97273b5ddce19f\": container with ID starting with 6d6ab5611ab9b5e9766edbe1bc297fea9b7374ed8823260c6a97273b5ddce19f not found: ID does not exist"
Nov 03 20:24:15.339999 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:15.339946    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5" podStartSLOduration=2.3399284160000002 podStartE2EDuration="2.339928416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:15.335468381 +0000 UTC m=+1385.468596843" watchObservedRunningTime="2026-11-03 20:24:15.339928416 +0000 UTC m=+1385.473056882"
Nov 03 20:24:15.380284 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:15.380252    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj"]
Nov 03 20:24:15.388511 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:15.388484    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1b646f-8686c6c6b8-sj8kj"]
Nov 03 20:24:15.927639 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:15.926716    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:15.993199 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:15.992905    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:16.033983 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:16.033944    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5" path="/var/lib/kubelet/pods/a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5/volumes"
Nov 03 20:24:16.385625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:16.383217    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:16.385625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:16.384336    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:19.630097 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:24:19.630010    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:24:20.245732 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.245703    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl"]
Nov 03 20:24:20.246325 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.246312    4421 cpu_manager.go:414] "containerMap: removing container" podUID="f8732447-9533-40b3-8da9-e8d6d946785e" containerName="istio-proxy"
Nov 03 20:24:20.246381 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.246327    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8732447-9533-40b3-8da9-e8d6d946785e" containerName="istio-proxy"
Nov 03 20:24:20.246381 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.246367    4421 cpu_manager.go:414] "containerMap: removing container" podUID="a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5" containerName="payload-processing"
Nov 03 20:24:20.246381 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.246371    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5" containerName="payload-processing"
Nov 03 20:24:20.246469 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.246457    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c263b0e4-5dd7-4a46-a527-f461b779e42f" containerName="istio-proxy"
Nov 03 20:24:20.246469 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.246466    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4eb3e18-16aa-4c22-92c2-c3dd2c1ac3d5" containerName="payload-processing"
Nov 03 20:24:20.246533 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.246472    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8732447-9533-40b3-8da9-e8d6d946785e" containerName="istio-proxy"
Nov 03 20:24:20.246533 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.246487    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="436959f1-0deb-43cc-b8e0-ca75a6f0afbd" containerName="istio-proxy"
Nov 03 20:24:20.260700 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.260674    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl"]
Nov 03 20:24:20.260867 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.260809    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl"
Nov 03 20:24:20.265132 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.264860    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-59bc3b\""
Nov 03 20:24:20.265851 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.265813    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-59bc3b-dockercfg-8jkkc\""
Nov 03 20:24:20.293101 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.293077    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb"]
Nov 03 20:24:20.322262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.297423    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb"
Nov 03 20:24:20.322262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.306525    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb"]
Nov 03 20:24:20.413631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.365552    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1469175-be50-4567-9235-31dd037fd284-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb\" (UID: \"b1469175-be50-4567-9235-31dd037fd284\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb"
Nov 03 20:24:20.413631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.365750    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbtgf\" (UniqueName: \"kubernetes.io/projected/cf8fcf01-c9dd-4a54-8d17-8986f7f27647-kube-api-access-hbtgf\") pod \"payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl\" (UID: \"cf8fcf01-c9dd-4a54-8d17-8986f7f27647\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl"
Nov 03 20:24:20.413631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.365891    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqmzl\" (UniqueName: \"kubernetes.io/projected/b1469175-be50-4567-9235-31dd037fd284-kube-api-access-jqmzl\") pod \"payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb\" (UID: \"b1469175-be50-4567-9235-31dd037fd284\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb"
Nov 03 20:24:20.413631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.365954    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8fcf01-c9dd-4a54-8d17-8986f7f27647-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl\" (UID: \"cf8fcf01-c9dd-4a54-8d17-8986f7f27647\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl"
Nov 03 20:24:20.468899 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.468850    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8fcf01-c9dd-4a54-8d17-8986f7f27647-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl\" (UID: \"cf8fcf01-c9dd-4a54-8d17-8986f7f27647\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl"
Nov 03 20:24:20.509428 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.509330    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1469175-be50-4567-9235-31dd037fd284-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb\" (UID: \"b1469175-be50-4567-9235-31dd037fd284\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb"
Nov 03 20:24:20.511579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.511552    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbtgf\" (UniqueName: \"kubernetes.io/projected/cf8fcf01-c9dd-4a54-8d17-8986f7f27647-kube-api-access-hbtgf\") pod \"payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl\" (UID: \"cf8fcf01-c9dd-4a54-8d17-8986f7f27647\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl"
Nov 03 20:24:20.513101 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.513074    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmzl\" (UniqueName: \"kubernetes.io/projected/b1469175-be50-4567-9235-31dd037fd284-kube-api-access-jqmzl\") pod \"payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb\" (UID: \"b1469175-be50-4567-9235-31dd037fd284\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb"
Nov 03 20:24:20.513352 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.473863    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8fcf01-c9dd-4a54-8d17-8986f7f27647-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl\" (UID: \"cf8fcf01-c9dd-4a54-8d17-8986f7f27647\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl"
Nov 03 20:24:20.513458 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.511188    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1469175-be50-4567-9235-31dd037fd284-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb\" (UID: \"b1469175-be50-4567-9235-31dd037fd284\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb"
Nov 03 20:24:20.570134 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.543025    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbtgf\" (UniqueName: \"kubernetes.io/projected/cf8fcf01-c9dd-4a54-8d17-8986f7f27647-kube-api-access-hbtgf\") pod \"payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl\" (UID: \"cf8fcf01-c9dd-4a54-8d17-8986f7f27647\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl"
Nov 03 20:24:20.570134 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.545573    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqmzl\" (UniqueName: \"kubernetes.io/projected/b1469175-be50-4567-9235-31dd037fd284-kube-api-access-jqmzl\") pod \"payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb\" (UID: \"b1469175-be50-4567-9235-31dd037fd284\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb"
Nov 03 20:24:20.580367 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.580069    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl"
Nov 03 20:24:20.626654 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.626589    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb"
Nov 03 20:24:20.882850 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.882821    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl"]
Nov 03 20:24:20.883375 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:24:20.883350    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf8fcf01_c9dd_4a54_8d17_8986f7f27647.slice/crio-24ba4caaeac0ce91bdd95efe485762db6352faa0cb3bf4998e56140594241476 WatchSource:0}: Error finding container 24ba4caaeac0ce91bdd95efe485762db6352faa0cb3bf4998e56140594241476: Status 404 returned error can't find the container with id 24ba4caaeac0ce91bdd95efe485762db6352faa0cb3bf4998e56140594241476
Nov 03 20:24:20.940054 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:24:20.940026    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1469175_be50_4567_9235_31dd037fd284.slice/crio-beeb17d27d0f92fef578e2ce5615ef5c820cf8d3c28222fda7308d667ab2be07 WatchSource:0}: Error finding container beeb17d27d0f92fef578e2ce5615ef5c820cf8d3c28222fda7308d667ab2be07: Status 404 returned error can't find the container with id beeb17d27d0f92fef578e2ce5615ef5c820cf8d3c28222fda7308d667ab2be07
Nov 03 20:24:20.940516 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:20.940495    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb"]
Nov 03 20:24:21.055217 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.055182    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p"]
Nov 03 20:24:21.055455 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.055426    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p" podUID="4dba43a0-1306-49e2-8f2c-bc5e221381d2" containerName="maas-api" containerID="cri-o://d1fc3c72f0ae388bc87f0a235850bc56aa434ff11fcd0526cf93c579a972b755" gracePeriod=30
Nov 03 20:24:21.485590 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.485557    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p"
Nov 03 20:24:21.518935 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.518895    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl" event={"ID":"cf8fcf01-c9dd-4a54-8d17-8986f7f27647","Type":"ContainerStarted","Data":"4081179fc2ae7ac39be6bddbde840faac96f13e327d7ea6bbf332bb753e9c966"}
Nov 03 20:24:21.519122 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.518987    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl" event={"ID":"cf8fcf01-c9dd-4a54-8d17-8986f7f27647","Type":"ContainerStarted","Data":"24ba4caaeac0ce91bdd95efe485762db6352faa0cb3bf4998e56140594241476"}
Nov 03 20:24:21.519348 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.519309    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl"
Nov 03 20:24:21.523907 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.523877    4421 generic.go:356] "Generic (PLEG): container finished" podID="aab9be1c-04f0-4285-9e31-fbb87ec0da02" containerID="9e1a289410f2541b26b444264e777710cda813c5c8a70e6829a3b6aa0caced76" exitCode=0
Nov 03 20:24:21.524033 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.523991    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz" event={"ID":"aab9be1c-04f0-4285-9e31-fbb87ec0da02","Type":"ContainerDied","Data":"9e1a289410f2541b26b444264e777710cda813c5c8a70e6829a3b6aa0caced76"}
Nov 03 20:24:21.524487 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.524467    4421 scope.go:122] "RemoveContainer" containerID="9e1a289410f2541b26b444264e777710cda813c5c8a70e6829a3b6aa0caced76"
Nov 03 20:24:21.526544 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.526436    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:24:21.526544 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.526516    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:24:21.526732 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.526552    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:24:21.527567 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.527549    4421 generic.go:356] "Generic (PLEG): container finished" podID="4dba43a0-1306-49e2-8f2c-bc5e221381d2" containerID="d1fc3c72f0ae388bc87f0a235850bc56aa434ff11fcd0526cf93c579a972b755" exitCode=0
Nov 03 20:24:21.527733 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.527710    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p"
Nov 03 20:24:21.527807 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.527769    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p" event={"ID":"4dba43a0-1306-49e2-8f2c-bc5e221381d2","Type":"ContainerDied","Data":"d1fc3c72f0ae388bc87f0a235850bc56aa434ff11fcd0526cf93c579a972b755"}
Nov 03 20:24:21.527871 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.527816    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p" event={"ID":"4dba43a0-1306-49e2-8f2c-bc5e221381d2","Type":"ContainerDied","Data":"c4a64a64dff61a8cd303f08fa4c225887b1e9655cfcec4965ac2ba54ad52a34a"}
Nov 03 20:24:21.527871 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.527846    4421 scope.go:122] "RemoveContainer" containerID="d1fc3c72f0ae388bc87f0a235850bc56aa434ff11fcd0526cf93c579a972b755"
Nov 03 20:24:21.531004 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.530982    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb" event={"ID":"b1469175-be50-4567-9235-31dd037fd284","Type":"ContainerStarted","Data":"6cdd6b373f8eecba75c960b67b36fa7b9aa7f2c8d4e4a2468d06837be36c9440"}
Nov 03 20:24:21.531095 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.531025    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb" event={"ID":"b1469175-be50-4567-9235-31dd037fd284","Type":"ContainerStarted","Data":"beeb17d27d0f92fef578e2ce5615ef5c820cf8d3c28222fda7308d667ab2be07"}
Nov 03 20:24:21.531228 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.531213    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb"
Nov 03 20:24:21.538910 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.538850    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl" podStartSLOduration=1.538834007 podStartE2EDuration="1.538834007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:21.537137362 +0000 UTC m=+1391.670265835" watchObservedRunningTime="2026-11-03 20:24:21.538834007 +0000 UTC m=+1391.671962465"
Nov 03 20:24:21.576180 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.576128    4421 scope.go:122] "RemoveContainer" containerID="d1fc3c72f0ae388bc87f0a235850bc56aa434ff11fcd0526cf93c579a972b755"
Nov 03 20:24:21.576647 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:24:21.576611    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1fc3c72f0ae388bc87f0a235850bc56aa434ff11fcd0526cf93c579a972b755\": container with ID starting with d1fc3c72f0ae388bc87f0a235850bc56aa434ff11fcd0526cf93c579a972b755 not found: ID does not exist" containerID="d1fc3c72f0ae388bc87f0a235850bc56aa434ff11fcd0526cf93c579a972b755"
Nov 03 20:24:21.576770 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.576650    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1fc3c72f0ae388bc87f0a235850bc56aa434ff11fcd0526cf93c579a972b755"} err="failed to get container status \"d1fc3c72f0ae388bc87f0a235850bc56aa434ff11fcd0526cf93c579a972b755\": rpc error: code = NotFound desc = could not find container \"d1fc3c72f0ae388bc87f0a235850bc56aa434ff11fcd0526cf93c579a972b755\": container with ID starting with d1fc3c72f0ae388bc87f0a235850bc56aa434ff11fcd0526cf93c579a972b755 not found: ID does not exist"
Nov 03 20:24:21.601721 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.601586    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dba43a0-1306-49e2-8f2c-bc5e221381d2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dba43a0-1306-49e2-8f2c-bc5e221381d2-maas-api-tls\") pod \"4dba43a0-1306-49e2-8f2c-bc5e221381d2\" (UID: \"4dba43a0-1306-49e2-8f2c-bc5e221381d2\") "
Nov 03 20:24:21.602183 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.602157    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dba43a0-1306-49e2-8f2c-bc5e221381d2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dba43a0-1306-49e2-8f2c-bc5e221381d2-metrics-certs\") pod \"4dba43a0-1306-49e2-8f2c-bc5e221381d2\" (UID: \"4dba43a0-1306-49e2-8f2c-bc5e221381d2\") "
Nov 03 20:24:21.602401 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.602384    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dba43a0-1306-49e2-8f2c-bc5e221381d2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dba43a0-1306-49e2-8f2c-bc5e221381d2-openshift-service-ca\") pod \"4dba43a0-1306-49e2-8f2c-bc5e221381d2\" (UID: \"4dba43a0-1306-49e2-8f2c-bc5e221381d2\") "
Nov 03 20:24:21.602590 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.602575    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dba43a0-1306-49e2-8f2c-bc5e221381d2-kube-api-access-dwjfc\" (UniqueName: \"kubernetes.io/projected/4dba43a0-1306-49e2-8f2c-bc5e221381d2-kube-api-access-dwjfc\") pod \"4dba43a0-1306-49e2-8f2c-bc5e221381d2\" (UID: \"4dba43a0-1306-49e2-8f2c-bc5e221381d2\") "
Nov 03 20:24:21.606208 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.605217    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dba43a0-1306-49e2-8f2c-bc5e221381d2-openshift-service-ca" pod "4dba43a0-1306-49e2-8f2c-bc5e221381d2" (UID: "4dba43a0-1306-49e2-8f2c-bc5e221381d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:24:21.610869 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.610840    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dba43a0-1306-49e2-8f2c-bc5e221381d2-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:21.617509 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.616477    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb" podStartSLOduration=1.616438895 podStartE2EDuration="1.616438895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:21.605349223 +0000 UTC m=+1391.738477686" watchObservedRunningTime="2026-11-03 20:24:21.616438895 +0000 UTC m=+1391.749567357"
Nov 03 20:24:21.631402 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.631374    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dba43a0-1306-49e2-8f2c-bc5e221381d2-kube-api-access-dwjfc" pod "4dba43a0-1306-49e2-8f2c-bc5e221381d2" (UID: "4dba43a0-1306-49e2-8f2c-bc5e221381d2"). InnerVolumeSpecName "kube-api-access-dwjfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:24:21.633580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.633504    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dba43a0-1306-49e2-8f2c-bc5e221381d2-maas-api-tls" pod "4dba43a0-1306-49e2-8f2c-bc5e221381d2" (UID: "4dba43a0-1306-49e2-8f2c-bc5e221381d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:24:21.634960 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.634935    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dba43a0-1306-49e2-8f2c-bc5e221381d2-metrics-certs" pod "4dba43a0-1306-49e2-8f2c-bc5e221381d2" (UID: "4dba43a0-1306-49e2-8f2c-bc5e221381d2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:24:21.750759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.725016    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dba43a0-1306-49e2-8f2c-bc5e221381d2-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:21.751037 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.751011    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dba43a0-1306-49e2-8f2c-bc5e221381d2-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:21.751153 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.751137    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwjfc\" (UniqueName: \"kubernetes.io/projected/4dba43a0-1306-49e2-8f2c-bc5e221381d2-kube-api-access-dwjfc\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:21.856637 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.856572    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p"]
Nov 03 20:24:21.859912 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:21.859889    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1b646f-f6bd76f88-twl8p"]
Nov 03 20:24:22.017852 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:22.017768    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dba43a0-1306-49e2-8f2c-bc5e221381d2" path="/var/lib/kubelet/pods/4dba43a0-1306-49e2-8f2c-bc5e221381d2/volumes"
Nov 03 20:24:22.540216 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:22.540183    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz" event={"ID":"aab9be1c-04f0-4285-9e31-fbb87ec0da02","Type":"ContainerStarted","Data":"87fa026b76b34a2cb0fc2846f966c1c69fa5d9758e8fb5aae830a7aa31dec49a"}
Nov 03 20:24:23.096232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:23.096196    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:24:23.097354 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:23.097315    4421 patch_prober.go:29] interesting pod/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 03 20:24:23.097495 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:23.097380    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz" podUID="aab9be1c-04f0-4285-9e31-fbb87ec0da02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 03 20:24:24.096164 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:24.096126    4421 patch_prober.go:29] interesting pod/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 03 20:24:24.096456 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:24.096424    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz" podUID="aab9be1c-04f0-4285-9e31-fbb87ec0da02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 03 20:24:25.096362 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:25.096326    4421 patch_prober.go:29] interesting pod/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 03 20:24:25.096569 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:25.096389    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz" podUID="aab9be1c-04f0-4285-9e31-fbb87ec0da02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 03 20:24:25.398446 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:25.398359    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"]
Nov 03 20:24:25.398982 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:25.398949    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz" podUID="aab9be1c-04f0-4285-9e31-fbb87ec0da02" containerName="istio-proxy" containerID="cri-o://87fa026b76b34a2cb0fc2846f966c1c69fa5d9758e8fb5aae830a7aa31dec49a" gracePeriod=30
Nov 03 20:24:25.505880 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:25.505729    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:24:25.830477 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:25.830445    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r"]
Nov 03 20:24:25.830910 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:25.830884    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r" podUID="b319a32a-24a1-4380-aa49-d0ac98924905" containerName="maas-api" containerID="cri-o://bebd8712575b2f22ebe6279c8e588f84919a5d2c055bb0386fcacceba39797ed" gracePeriod=30
Nov 03 20:24:26.389570 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:26.389536    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r"
Nov 03 20:24:26.487342 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:26.487298    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b319a32a-24a1-4380-aa49-d0ac98924905-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b319a32a-24a1-4380-aa49-d0ac98924905-metrics-certs\") pod \"b319a32a-24a1-4380-aa49-d0ac98924905\" (UID: \"b319a32a-24a1-4380-aa49-d0ac98924905\") "
Nov 03 20:24:26.487641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:26.487619    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b319a32a-24a1-4380-aa49-d0ac98924905-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b319a32a-24a1-4380-aa49-d0ac98924905-maas-api-tls\") pod \"b319a32a-24a1-4380-aa49-d0ac98924905\" (UID: \"b319a32a-24a1-4380-aa49-d0ac98924905\") "
Nov 03 20:24:26.487812 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:26.487796    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b319a32a-24a1-4380-aa49-d0ac98924905-kube-api-access-nvz6k\" (UniqueName: \"kubernetes.io/projected/b319a32a-24a1-4380-aa49-d0ac98924905-kube-api-access-nvz6k\") pod \"b319a32a-24a1-4380-aa49-d0ac98924905\" (UID: \"b319a32a-24a1-4380-aa49-d0ac98924905\") "
Nov 03 20:24:26.488131 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:26.488107    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b319a32a-24a1-4380-aa49-d0ac98924905-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b319a32a-24a1-4380-aa49-d0ac98924905-openshift-service-ca\") pod \"b319a32a-24a1-4380-aa49-d0ac98924905\" (UID: \"b319a32a-24a1-4380-aa49-d0ac98924905\") "
Nov 03 20:24:26.495424 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:26.495389    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b319a32a-24a1-4380-aa49-d0ac98924905-openshift-service-ca" pod "b319a32a-24a1-4380-aa49-d0ac98924905" (UID: "b319a32a-24a1-4380-aa49-d0ac98924905"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:24:26.498324 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:26.498276    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b319a32a-24a1-4380-aa49-d0ac98924905-kube-api-access-nvz6k" pod "b319a32a-24a1-4380-aa49-d0ac98924905" (UID: "b319a32a-24a1-4380-aa49-d0ac98924905"). InnerVolumeSpecName "kube-api-access-nvz6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:24:26.500079 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:26.500046    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b319a32a-24a1-4380-aa49-d0ac98924905-metrics-certs" pod "b319a32a-24a1-4380-aa49-d0ac98924905" (UID: "b319a32a-24a1-4380-aa49-d0ac98924905"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:24:26.500337 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:26.500314    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b319a32a-24a1-4380-aa49-d0ac98924905-maas-api-tls" pod "b319a32a-24a1-4380-aa49-d0ac98924905" (UID: "b319a32a-24a1-4380-aa49-d0ac98924905"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:24:26.590350 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:26.590258    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b319a32a-24a1-4380-aa49-d0ac98924905-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:26.590350 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:26.590290    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b319a32a-24a1-4380-aa49-d0ac98924905-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:26.590350 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:26.590300    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvz6k\" (UniqueName: \"kubernetes.io/projected/b319a32a-24a1-4380-aa49-d0ac98924905-kube-api-access-nvz6k\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:26.590350 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:26.590310    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b319a32a-24a1-4380-aa49-d0ac98924905-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:26.678409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:26.678362    4421 generic.go:356] "Generic (PLEG): container finished" podID="b319a32a-24a1-4380-aa49-d0ac98924905" containerID="bebd8712575b2f22ebe6279c8e588f84919a5d2c055bb0386fcacceba39797ed" exitCode=0
Nov 03 20:24:26.678409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:26.678410    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r"
Nov 03 20:24:26.678684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:26.678443    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r" event={"ID":"b319a32a-24a1-4380-aa49-d0ac98924905","Type":"ContainerDied","Data":"bebd8712575b2f22ebe6279c8e588f84919a5d2c055bb0386fcacceba39797ed"}
Nov 03 20:24:26.678684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:26.678502    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r" event={"ID":"b319a32a-24a1-4380-aa49-d0ac98924905","Type":"ContainerDied","Data":"0bca35ff66bb64b981c914d8df7ca838439faa2cb43c0bac14c1f5276e5c51fb"}
Nov 03 20:24:26.678684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:26.678523    4421 scope.go:122] "RemoveContainer" containerID="bebd8712575b2f22ebe6279c8e588f84919a5d2c055bb0386fcacceba39797ed"
Nov 03 20:24:26.689876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:26.689856    4421 scope.go:122] "RemoveContainer" containerID="bebd8712575b2f22ebe6279c8e588f84919a5d2c055bb0386fcacceba39797ed"
Nov 03 20:24:26.690168 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:24:26.690150    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bebd8712575b2f22ebe6279c8e588f84919a5d2c055bb0386fcacceba39797ed\": container with ID starting with bebd8712575b2f22ebe6279c8e588f84919a5d2c055bb0386fcacceba39797ed not found: ID does not exist" containerID="bebd8712575b2f22ebe6279c8e588f84919a5d2c055bb0386fcacceba39797ed"
Nov 03 20:24:26.690216 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:26.690176    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bebd8712575b2f22ebe6279c8e588f84919a5d2c055bb0386fcacceba39797ed"} err="failed to get container status \"bebd8712575b2f22ebe6279c8e588f84919a5d2c055bb0386fcacceba39797ed\": rpc error: code = NotFound desc = could not find container \"bebd8712575b2f22ebe6279c8e588f84919a5d2c055bb0386fcacceba39797ed\": container with ID starting with bebd8712575b2f22ebe6279c8e588f84919a5d2c055bb0386fcacceba39797ed not found: ID does not exist"
Nov 03 20:24:26.703359 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:26.703326    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r"]
Nov 03 20:24:26.706940 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:26.706919    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9e5029-9c89b5667-9hd5r"]
Nov 03 20:24:27.919113 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:27.919073    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9"]
Nov 03 20:24:27.919932 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:27.919913    4421 cpu_manager.go:414] "containerMap: removing container" podUID="4dba43a0-1306-49e2-8f2c-bc5e221381d2" containerName="maas-api"
Nov 03 20:24:27.919932 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:27.919933    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dba43a0-1306-49e2-8f2c-bc5e221381d2" containerName="maas-api"
Nov 03 20:24:27.920123 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:27.919961    4421 cpu_manager.go:414] "containerMap: removing container" podUID="b319a32a-24a1-4380-aa49-d0ac98924905" containerName="maas-api"
Nov 03 20:24:27.920123 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:27.919969    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b319a32a-24a1-4380-aa49-d0ac98924905" containerName="maas-api"
Nov 03 20:24:27.920235 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:27.920156    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dba43a0-1306-49e2-8f2c-bc5e221381d2" containerName="maas-api"
Nov 03 20:24:27.920235 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:27.920173    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="b319a32a-24a1-4380-aa49-d0ac98924905" containerName="maas-api"
Nov 03 20:24:27.923186 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:27.923166    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9"
Nov 03 20:24:27.931330 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:27.931223    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-59bc3b-serving-cert\""
Nov 03 20:24:27.973699 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:27.973668    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9"]
Nov 03 20:24:28.010794 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:28.010760    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58d8fae7-6dba-491c-860e-05d2ec4e08f2-maas-api-tls\") pod \"maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9\" (UID: \"58d8fae7-6dba-491c-860e-05d2ec4e08f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9"
Nov 03 20:24:28.010979 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:28.010880    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf276\" (UniqueName: \"kubernetes.io/projected/58d8fae7-6dba-491c-860e-05d2ec4e08f2-kube-api-access-mf276\") pod \"maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9\" (UID: \"58d8fae7-6dba-491c-860e-05d2ec4e08f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9"
Nov 03 20:24:28.011068 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:28.011052    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58d8fae7-6dba-491c-860e-05d2ec4e08f2-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9\" (UID: \"58d8fae7-6dba-491c-860e-05d2ec4e08f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9"
Nov 03 20:24:28.011115 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:28.011102    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58d8fae7-6dba-491c-860e-05d2ec4e08f2-metrics-certs\") pod \"maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9\" (UID: \"58d8fae7-6dba-491c-860e-05d2ec4e08f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9"
Nov 03 20:24:28.016934 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:28.016908    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b319a32a-24a1-4380-aa49-d0ac98924905" path="/var/lib/kubelet/pods/b319a32a-24a1-4380-aa49-d0ac98924905/volumes"
Nov 03 20:24:28.112541 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:28.112510    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58d8fae7-6dba-491c-860e-05d2ec4e08f2-maas-api-tls\") pod \"maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9\" (UID: \"58d8fae7-6dba-491c-860e-05d2ec4e08f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9"
Nov 03 20:24:28.112765 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:28.112582    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf276\" (UniqueName: \"kubernetes.io/projected/58d8fae7-6dba-491c-860e-05d2ec4e08f2-kube-api-access-mf276\") pod \"maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9\" (UID: \"58d8fae7-6dba-491c-860e-05d2ec4e08f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9"
Nov 03 20:24:28.112765 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:28.112753    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58d8fae7-6dba-491c-860e-05d2ec4e08f2-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9\" (UID: \"58d8fae7-6dba-491c-860e-05d2ec4e08f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9"
Nov 03 20:24:28.112892 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:28.112810    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58d8fae7-6dba-491c-860e-05d2ec4e08f2-metrics-certs\") pod \"maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9\" (UID: \"58d8fae7-6dba-491c-860e-05d2ec4e08f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9"
Nov 03 20:24:28.113799 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:28.113739    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58d8fae7-6dba-491c-860e-05d2ec4e08f2-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9\" (UID: \"58d8fae7-6dba-491c-860e-05d2ec4e08f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9"
Nov 03 20:24:28.119405 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:28.115785    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58d8fae7-6dba-491c-860e-05d2ec4e08f2-metrics-certs\") pod \"maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9\" (UID: \"58d8fae7-6dba-491c-860e-05d2ec4e08f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9"
Nov 03 20:24:28.119405 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:28.119307    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58d8fae7-6dba-491c-860e-05d2ec4e08f2-maas-api-tls\") pod \"maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9\" (UID: \"58d8fae7-6dba-491c-860e-05d2ec4e08f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9"
Nov 03 20:24:28.178675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:28.176765    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf276\" (UniqueName: \"kubernetes.io/projected/58d8fae7-6dba-491c-860e-05d2ec4e08f2-kube-api-access-mf276\") pod \"maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9\" (UID: \"58d8fae7-6dba-491c-860e-05d2ec4e08f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9"
Nov 03 20:24:28.283622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:28.282834    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9"
Nov 03 20:24:28.671632 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:28.671071    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9"]
Nov 03 20:24:28.707664 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:28.700370    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9" event={"ID":"58d8fae7-6dba-491c-860e-05d2ec4e08f2","Type":"ContainerStarted","Data":"1b1afeb51ef44979401d95ccdbee34966dd9477a31e8c5f1daead70f7b78fcb5"}
Nov 03 20:24:29.710892 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:29.710856    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9" event={"ID":"58d8fae7-6dba-491c-860e-05d2ec4e08f2","Type":"ContainerStarted","Data":"2046101b0a3e1af688613c54ecd7e4777a276e0e6dd1bdebb6b13feb2a15c2ac"}
Nov 03 20:24:29.711271 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:29.710990    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9"
Nov 03 20:24:29.735845 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:29.735790    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9" podStartSLOduration=2.102216568 podStartE2EDuration="2.735775116s" totalImagesPullingTime="633.558548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:29.733039407 +0000 UTC m=+1399.866167871" watchObservedRunningTime="2026-11-03 20:24:29.735775116 +0000 UTC m=+1399.868903588"
Nov 03 20:24:32.566631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:32.565923    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl"
Nov 03 20:24:32.566631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:32.566113    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb"
Nov 03 20:24:33.580381 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.580349    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:24:33.671768 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.671681    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-credential-socket\") pod \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") "
Nov 03 20:24:33.671768 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.671719    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aab9be1c-04f0-4285-9e31-fbb87ec0da02-kube-api-access-w7gzc\" (UniqueName: \"kubernetes.io/projected/aab9be1c-04f0-4285-9e31-fbb87ec0da02-kube-api-access-w7gzc\") pod \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") "
Nov 03 20:24:33.672006 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.671825    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-token\" (UniqueName: \"kubernetes.io/projected/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-token\") pod \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") "
Nov 03 20:24:33.672006 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.671914    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-podinfo\") pod \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") "
Nov 03 20:24:33.672006 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.671936    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istiod-ca-cert\") pod \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") "
Nov 03 20:24:33.672006 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.671964    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-workload-socket\") pod \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") "
Nov 03 20:24:33.672006 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.671984    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-envoy\") pod \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") "
Nov 03 20:24:33.672267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.672030    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-data\") pod \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") "
Nov 03 20:24:33.672267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.672065    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-workload-certs\") pod \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\" (UID: \"aab9be1c-04f0-4285-9e31-fbb87ec0da02\") "
Nov 03 20:24:33.672383 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.672008    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-credential-socket" pod "aab9be1c-04f0-4285-9e31-fbb87ec0da02" (UID: "aab9be1c-04f0-4285-9e31-fbb87ec0da02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:24:33.672383 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.672277    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-workload-socket" pod "aab9be1c-04f0-4285-9e31-fbb87ec0da02" (UID: "aab9be1c-04f0-4285-9e31-fbb87ec0da02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:24:33.672383 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.672361    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istiod-ca-cert" pod "aab9be1c-04f0-4285-9e31-fbb87ec0da02" (UID: "aab9be1c-04f0-4285-9e31-fbb87ec0da02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:24:33.672546 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.672485    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-data" pod "aab9be1c-04f0-4285-9e31-fbb87ec0da02" (UID: "aab9be1c-04f0-4285-9e31-fbb87ec0da02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:24:33.672621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.672549    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-workload-certs" pod "aab9be1c-04f0-4285-9e31-fbb87ec0da02" (UID: "aab9be1c-04f0-4285-9e31-fbb87ec0da02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:24:33.672819 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.672802    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:33.672898 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.672823    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:33.672898 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.672837    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:33.672898 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.672849    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:33.672898 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.672862    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:33.674332 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.674308    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab9be1c-04f0-4285-9e31-fbb87ec0da02-kube-api-access-w7gzc" pod "aab9be1c-04f0-4285-9e31-fbb87ec0da02" (UID: "aab9be1c-04f0-4285-9e31-fbb87ec0da02"). InnerVolumeSpecName "kube-api-access-w7gzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:24:33.674663 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.674637    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-podinfo" pod "aab9be1c-04f0-4285-9e31-fbb87ec0da02" (UID: "aab9be1c-04f0-4285-9e31-fbb87ec0da02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:24:33.674770 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.674672    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-token" pod "aab9be1c-04f0-4285-9e31-fbb87ec0da02" (UID: "aab9be1c-04f0-4285-9e31-fbb87ec0da02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:24:33.674770 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.674689    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-envoy" pod "aab9be1c-04f0-4285-9e31-fbb87ec0da02" (UID: "aab9be1c-04f0-4285-9e31-fbb87ec0da02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:24:33.741914 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.741884    4421 generic.go:356] "Generic (PLEG): container finished" podID="aab9be1c-04f0-4285-9e31-fbb87ec0da02" containerID="87fa026b76b34a2cb0fc2846f966c1c69fa5d9758e8fb5aae830a7aa31dec49a" exitCode=0
Nov 03 20:24:33.742058 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.741942    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"
Nov 03 20:24:33.742058 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.741976    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz" event={"ID":"aab9be1c-04f0-4285-9e31-fbb87ec0da02","Type":"ContainerDied","Data":"87fa026b76b34a2cb0fc2846f966c1c69fa5d9758e8fb5aae830a7aa31dec49a"}
Nov 03 20:24:33.742058 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.742042    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz" event={"ID":"aab9be1c-04f0-4285-9e31-fbb87ec0da02","Type":"ContainerDied","Data":"187eb0cc0e4e9b18b2170ec46365099b69cd6872736da9a5dc98a4a64dbef82c"}
Nov 03 20:24:33.742244 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.742071    4421 scope.go:122] "RemoveContainer" containerID="87fa026b76b34a2cb0fc2846f966c1c69fa5d9758e8fb5aae830a7aa31dec49a"
Nov 03 20:24:33.753746 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.753732    4421 scope.go:122] "RemoveContainer" containerID="9e1a289410f2541b26b444264e777710cda813c5c8a70e6829a3b6aa0caced76"
Nov 03 20:24:33.763317 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.763300    4421 scope.go:122] "RemoveContainer" containerID="87fa026b76b34a2cb0fc2846f966c1c69fa5d9758e8fb5aae830a7aa31dec49a"
Nov 03 20:24:33.763842 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:24:33.763822    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87fa026b76b34a2cb0fc2846f966c1c69fa5d9758e8fb5aae830a7aa31dec49a\": container with ID starting with 87fa026b76b34a2cb0fc2846f966c1c69fa5d9758e8fb5aae830a7aa31dec49a not found: ID does not exist" containerID="87fa026b76b34a2cb0fc2846f966c1c69fa5d9758e8fb5aae830a7aa31dec49a"
Nov 03 20:24:33.763952 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.763849    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87fa026b76b34a2cb0fc2846f966c1c69fa5d9758e8fb5aae830a7aa31dec49a"} err="failed to get container status \"87fa026b76b34a2cb0fc2846f966c1c69fa5d9758e8fb5aae830a7aa31dec49a\": rpc error: code = NotFound desc = could not find container \"87fa026b76b34a2cb0fc2846f966c1c69fa5d9758e8fb5aae830a7aa31dec49a\": container with ID starting with 87fa026b76b34a2cb0fc2846f966c1c69fa5d9758e8fb5aae830a7aa31dec49a not found: ID does not exist"
Nov 03 20:24:33.763952 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.763876    4421 scope.go:122] "RemoveContainer" containerID="9e1a289410f2541b26b444264e777710cda813c5c8a70e6829a3b6aa0caced76"
Nov 03 20:24:33.764163 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:24:33.764129    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e1a289410f2541b26b444264e777710cda813c5c8a70e6829a3b6aa0caced76\": container with ID starting with 9e1a289410f2541b26b444264e777710cda813c5c8a70e6829a3b6aa0caced76 not found: ID does not exist" containerID="9e1a289410f2541b26b444264e777710cda813c5c8a70e6829a3b6aa0caced76"
Nov 03 20:24:33.764219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.764173    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e1a289410f2541b26b444264e777710cda813c5c8a70e6829a3b6aa0caced76"} err="failed to get container status \"9e1a289410f2541b26b444264e777710cda813c5c8a70e6829a3b6aa0caced76\": rpc error: code = NotFound desc = could not find container \"9e1a289410f2541b26b444264e777710cda813c5c8a70e6829a3b6aa0caced76\": container with ID starting with 9e1a289410f2541b26b444264e777710cda813c5c8a70e6829a3b6aa0caced76 not found: ID does not exist"
Nov 03 20:24:33.767182 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.767142    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"]
Nov 03 20:24:33.770918 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.770897    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-1b646f-openshift-default-b8c6768df-7ftjz"]
Nov 03 20:24:33.774365 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.774334    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7gzc\" (UniqueName: \"kubernetes.io/projected/aab9be1c-04f0-4285-9e31-fbb87ec0da02-kube-api-access-w7gzc\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:33.774365 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.774361    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:33.774483 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.774372    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:33.774483 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:33.774381    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab9be1c-04f0-4285-9e31-fbb87ec0da02-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:34.059699 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:34.052559    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aab9be1c-04f0-4285-9e31-fbb87ec0da02" path="/var/lib/kubelet/pods/aab9be1c-04f0-4285-9e31-fbb87ec0da02/volumes"
Nov 03 20:24:35.723728 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:35.723698    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9"
Nov 03 20:24:48.221314 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:48.221260    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:24:48.230070 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:48.230015    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9" podUID="7eefc668-8d72-4ee2-b047-91d13318ec49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:48.342740 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:48.342697    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:48.422636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:48.387937    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:48.548682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:48.548527    4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:48.582520 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:48.582471    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:48.681550 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:48.680895    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:48.789622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:48.788897    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 20:24:48.954624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:48.950748    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:24:48.954624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:48.950811    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng" podUID="9f633998-421c-4d6f-9528-94114a4ca6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:48.988623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:48.987555    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:24:49.635351 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:24:49.635300    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:24:53.284467 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:53.284406    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25"]
Nov 03 20:24:53.285222 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:53.284724    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25" podUID="675e46aa-b032-4274-90ba-9fdf30de5d57" containerName="maas-api" containerID="cri-o://bda6278d37d89201e36d2a8984b85008c21d807e4a08d2627d7923fdcaf66b50" gracePeriod=30
Nov 03 20:24:53.805163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:53.805135    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25"
Nov 03 20:24:53.973407 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:53.973366    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/675e46aa-b032-4274-90ba-9fdf30de5d57-metrics-certs\" (UniqueName: \"kubernetes.io/secret/675e46aa-b032-4274-90ba-9fdf30de5d57-metrics-certs\") pod \"675e46aa-b032-4274-90ba-9fdf30de5d57\" (UID: \"675e46aa-b032-4274-90ba-9fdf30de5d57\") "
Nov 03 20:24:53.975758 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:53.975722    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/675e46aa-b032-4274-90ba-9fdf30de5d57-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/675e46aa-b032-4274-90ba-9fdf30de5d57-maas-api-tls\") pod \"675e46aa-b032-4274-90ba-9fdf30de5d57\" (UID: \"675e46aa-b032-4274-90ba-9fdf30de5d57\") "
Nov 03 20:24:53.976666 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:53.976644    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/675e46aa-b032-4274-90ba-9fdf30de5d57-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/675e46aa-b032-4274-90ba-9fdf30de5d57-openshift-service-ca\") pod \"675e46aa-b032-4274-90ba-9fdf30de5d57\" (UID: \"675e46aa-b032-4274-90ba-9fdf30de5d57\") "
Nov 03 20:24:53.977801 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:53.977659    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/675e46aa-b032-4274-90ba-9fdf30de5d57-openshift-service-ca" pod "675e46aa-b032-4274-90ba-9fdf30de5d57" (UID: "675e46aa-b032-4274-90ba-9fdf30de5d57"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:24:53.978075 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:53.978045    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/675e46aa-b032-4274-90ba-9fdf30de5d57-kube-api-access-p8rxl\" (UniqueName: \"kubernetes.io/projected/675e46aa-b032-4274-90ba-9fdf30de5d57-kube-api-access-p8rxl\") pod \"675e46aa-b032-4274-90ba-9fdf30de5d57\" (UID: \"675e46aa-b032-4274-90ba-9fdf30de5d57\") "
Nov 03 20:24:53.980017 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:53.979985    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/675e46aa-b032-4274-90ba-9fdf30de5d57-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:54.004092 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:54.004032    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/675e46aa-b032-4274-90ba-9fdf30de5d57-metrics-certs" pod "675e46aa-b032-4274-90ba-9fdf30de5d57" (UID: "675e46aa-b032-4274-90ba-9fdf30de5d57"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:24:54.008923 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:54.005791    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/675e46aa-b032-4274-90ba-9fdf30de5d57-maas-api-tls" pod "675e46aa-b032-4274-90ba-9fdf30de5d57" (UID: "675e46aa-b032-4274-90ba-9fdf30de5d57"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:24:54.008923 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:54.007032    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/675e46aa-b032-4274-90ba-9fdf30de5d57-kube-api-access-p8rxl" pod "675e46aa-b032-4274-90ba-9fdf30de5d57" (UID: "675e46aa-b032-4274-90ba-9fdf30de5d57"). InnerVolumeSpecName "kube-api-access-p8rxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:24:54.081976 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:54.081944    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/675e46aa-b032-4274-90ba-9fdf30de5d57-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:54.081976 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:54.081978    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/675e46aa-b032-4274-90ba-9fdf30de5d57-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:54.082171 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:54.081993    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8rxl\" (UniqueName: \"kubernetes.io/projected/675e46aa-b032-4274-90ba-9fdf30de5d57-kube-api-access-p8rxl\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:24:54.178528 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:54.178496    4421 generic.go:356] "Generic (PLEG): container finished" podID="675e46aa-b032-4274-90ba-9fdf30de5d57" containerID="bda6278d37d89201e36d2a8984b85008c21d807e4a08d2627d7923fdcaf66b50" exitCode=0
Nov 03 20:24:54.178689 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:54.178543    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25"
Nov 03 20:24:54.178689 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:54.178587    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25" event={"ID":"675e46aa-b032-4274-90ba-9fdf30de5d57","Type":"ContainerDied","Data":"bda6278d37d89201e36d2a8984b85008c21d807e4a08d2627d7923fdcaf66b50"}
Nov 03 20:24:54.178689 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:54.178671    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25" event={"ID":"675e46aa-b032-4274-90ba-9fdf30de5d57","Type":"ContainerDied","Data":"e651a6dfcf624394cb6447eb9939616a5cb94d8393ff9adf62368af025f7e371"}
Nov 03 20:24:54.178863 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:54.178701    4421 scope.go:122] "RemoveContainer" containerID="bda6278d37d89201e36d2a8984b85008c21d807e4a08d2627d7923fdcaf66b50"
Nov 03 20:24:54.192816 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:54.192724    4421 scope.go:122] "RemoveContainer" containerID="bda6278d37d89201e36d2a8984b85008c21d807e4a08d2627d7923fdcaf66b50"
Nov 03 20:24:54.195100 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:24:54.195036    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bda6278d37d89201e36d2a8984b85008c21d807e4a08d2627d7923fdcaf66b50\": container with ID starting with bda6278d37d89201e36d2a8984b85008c21d807e4a08d2627d7923fdcaf66b50 not found: ID does not exist" containerID="bda6278d37d89201e36d2a8984b85008c21d807e4a08d2627d7923fdcaf66b50"
Nov 03 20:24:54.195100 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:54.195075    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bda6278d37d89201e36d2a8984b85008c21d807e4a08d2627d7923fdcaf66b50"} err="failed to get container status \"bda6278d37d89201e36d2a8984b85008c21d807e4a08d2627d7923fdcaf66b50\": rpc error: code = NotFound desc = could not find container \"bda6278d37d89201e36d2a8984b85008c21d807e4a08d2627d7923fdcaf66b50\": container with ID starting with bda6278d37d89201e36d2a8984b85008c21d807e4a08d2627d7923fdcaf66b50 not found: ID does not exist"
Nov 03 20:24:54.200380 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:54.200354    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25"]
Nov 03 20:24:54.209008 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:54.208984    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-019ef4-f5f6c549c-hfz25"]
Nov 03 20:24:56.280826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:56.280791    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="675e46aa-b032-4274-90ba-9fdf30de5d57" path="/var/lib/kubelet/pods/675e46aa-b032-4274-90ba-9fdf30de5d57/volumes"
Nov 03 20:24:58.294671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:58.294572    4421 generic.go:356] "Generic (PLEG): container finished" podID="8e6a0a0b-c9ca-4182-8962-fd4534a2844b" containerID="e563aea5ed0451dc85ae7b158bd943a4689594239571b31a5583cf922ea4f331" exitCode=0
Nov 03 20:24:58.294671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:58.294631    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5" event={"ID":"8e6a0a0b-c9ca-4182-8962-fd4534a2844b","Type":"ContainerDied","Data":"e563aea5ed0451dc85ae7b158bd943a4689594239571b31a5583cf922ea4f331"}
Nov 03 20:24:58.295099 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:58.295018    4421 scope.go:122] "RemoveContainer" containerID="e563aea5ed0451dc85ae7b158bd943a4689594239571b31a5583cf922ea4f331"
Nov 03 20:24:58.296580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:58.296548    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:24:58.296696 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:58.296678    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:24:58.296739 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:58.296723    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:24:59.304359 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:59.304327    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5" event={"ID":"8e6a0a0b-c9ca-4182-8962-fd4534a2844b","Type":"ContainerStarted","Data":"4f72288219f98fd4f341a1c01d4d0c0124b475d51d9a244a9c430be2715939f5"}
Nov 03 20:24:59.926073 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:59.926039    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:24:59.927464 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:59.927441    4421 patch_prober.go:29] interesting pod/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 20:24:59.927590 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:24:59.927493    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5" podUID="8e6a0a0b-c9ca-4182-8962-fd4534a2844b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 20:25:00.368745 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.368705    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"]
Nov 03 20:25:00.369922 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.369895    4421 cpu_manager.go:414] "containerMap: removing container" podUID="675e46aa-b032-4274-90ba-9fdf30de5d57" containerName="maas-api"
Nov 03 20:25:00.369922 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.369923    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="675e46aa-b032-4274-90ba-9fdf30de5d57" containerName="maas-api"
Nov 03 20:25:00.370085 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.369946    4421 cpu_manager.go:414] "containerMap: removing container" podUID="aab9be1c-04f0-4285-9e31-fbb87ec0da02" containerName="istio-proxy"
Nov 03 20:25:00.370085 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.369955    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aab9be1c-04f0-4285-9e31-fbb87ec0da02" containerName="istio-proxy"
Nov 03 20:25:00.370172 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.370117    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="675e46aa-b032-4274-90ba-9fdf30de5d57" containerName="maas-api"
Nov 03 20:25:00.370172 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.370136    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="aab9be1c-04f0-4285-9e31-fbb87ec0da02" containerName="istio-proxy"
Nov 03 20:25:00.370172 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.370147    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="aab9be1c-04f0-4285-9e31-fbb87ec0da02" containerName="istio-proxy"
Nov 03 20:25:00.373638 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.373576    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.383641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.382856    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"]
Nov 03 20:25:00.383810 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.383791    4421 cpu_manager.go:414] "containerMap: removing container" podUID="aab9be1c-04f0-4285-9e31-fbb87ec0da02" containerName="istio-proxy"
Nov 03 20:25:00.383871 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.383813    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aab9be1c-04f0-4285-9e31-fbb87ec0da02" containerName="istio-proxy"
Nov 03 20:25:00.389123 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.388520    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"]
Nov 03 20:25:00.389123 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.388662    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.513907 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.513875    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.514179 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.514154    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6cvq\" (UniqueName: \"kubernetes.io/projected/39b8d333-b40a-4096-9de9-a944842f9544-kube-api-access-t6cvq\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.514357 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.514338    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b8d333-b40a-4096-9de9-a944842f9544-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.514506 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.514489    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b8d333-b40a-4096-9de9-a944842f9544-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.514682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.514665    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.514838 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.514820    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.514983 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.514967    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.515092 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.515076    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75bkg\" (UniqueName: \"kubernetes.io/projected/57010672-22b7-4c7e-a3c0-75578515deb9-kube-api-access-75bkg\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.515206 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.515193    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.515444 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.515422    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57010672-22b7-4c7e-a3c0-75578515deb9-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.515627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.515587    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57010672-22b7-4c7e-a3c0-75578515deb9-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.515844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.515825    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.516361 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.516338    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.516536 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.516518    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.516702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.516686    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.516823 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.516805    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39b8d333-b40a-4096-9de9-a944842f9544-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.516931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.516916    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57010672-22b7-4c7e-a3c0-75578515deb9-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.517097 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.517080    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.519476 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.519458    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"]
Nov 03 20:25:00.618434 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.618403    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.618641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.618452    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.618641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.618480    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39b8d333-b40a-4096-9de9-a944842f9544-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.618641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.618497    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57010672-22b7-4c7e-a3c0-75578515deb9-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.618641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.618515    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.618641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.618542    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.618641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.618580    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6cvq\" (UniqueName: \"kubernetes.io/projected/39b8d333-b40a-4096-9de9-a944842f9544-kube-api-access-t6cvq\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.618641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.618631    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b8d333-b40a-4096-9de9-a944842f9544-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.618961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.618666    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b8d333-b40a-4096-9de9-a944842f9544-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.618961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.618706    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.618961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.618760    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.618961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.618791    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.618961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.618816    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75bkg\" (UniqueName: \"kubernetes.io/projected/57010672-22b7-4c7e-a3c0-75578515deb9-kube-api-access-75bkg\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.618961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.618843    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.618961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.618941    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.619337 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.618946    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.619337 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.618989    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57010672-22b7-4c7e-a3c0-75578515deb9-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.619337 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.619191    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.619337 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.619269    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.619337 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.619310    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57010672-22b7-4c7e-a3c0-75578515deb9-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.619782 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.619392    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.619782 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.619447    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.619908 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.619827    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.619994 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.619972    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57010672-22b7-4c7e-a3c0-75578515deb9-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.620099 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.620074    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.620171 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.620121    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.620546 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.620524    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.620770 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.620662    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b8d333-b40a-4096-9de9-a944842f9544-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.621703 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.621678    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.622203 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.622177    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57010672-22b7-4c7e-a3c0-75578515deb9-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.622341 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.622324    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.622723 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.622689    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b8d333-b40a-4096-9de9-a944842f9544-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.630983 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.630961    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6cvq\" (UniqueName: \"kubernetes.io/projected/39b8d333-b40a-4096-9de9-a944842f9544-kube-api-access-t6cvq\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.631273 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.631252    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39b8d333-b40a-4096-9de9-a944842f9544-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.631337 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.631312    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57010672-22b7-4c7e-a3c0-75578515deb9-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.632873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.632851    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75bkg\" (UniqueName: \"kubernetes.io/projected/57010672-22b7-4c7e-a3c0-75578515deb9-kube-api-access-75bkg\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.693474 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.693447    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:00.821367 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.821155    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:00.926483 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.926028    4421 patch_prober.go:29] interesting pod/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 20:25:00.926483 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:00.926085    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5" podUID="8e6a0a0b-c9ca-4182-8962-fd4534a2844b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 20:25:01.012639 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:01.012580    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"]
Nov 03 20:25:01.021403 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:25:01.021373    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b8d333_b40a_4096_9de9_a944842f9544.slice/crio-af1c12e2c885a64d22a2383253c22edfbd385442447513574bf83475f44306b6 WatchSource:0}: Error finding container af1c12e2c885a64d22a2383253c22edfbd385442447513574bf83475f44306b6: Status 404 returned error can't find the container with id af1c12e2c885a64d22a2383253c22edfbd385442447513574bf83475f44306b6
Nov 03 20:25:01.025155 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:01.025073    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:01.025278 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:01.025237    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:01.025582 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:01.025404    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:01.169831 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:01.169742    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"]
Nov 03 20:25:01.175209 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:25:01.175163    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57010672_22b7_4c7e_a3c0_75578515deb9.slice/crio-b4205acf6236434ac6e19178db2d5c88b76dce6514897f983bf61863fb5c9675 WatchSource:0}: Error finding container b4205acf6236434ac6e19178db2d5c88b76dce6514897f983bf61863fb5c9675: Status 404 returned error can't find the container with id b4205acf6236434ac6e19178db2d5c88b76dce6514897f983bf61863fb5c9675
Nov 03 20:25:01.185191 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:01.185142    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:01.185586 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:01.185541    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:01.185728 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:01.185682    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:01.351627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:01.341296    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87" event={"ID":"57010672-22b7-4c7e-a3c0-75578515deb9","Type":"ContainerStarted","Data":"443a3fcaeafe3d2e714a3b8165f476a77e184f3c9ad4d2ce7a43c1aa8eff8c3d"}
Nov 03 20:25:01.351627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:01.341412    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87" event={"ID":"57010672-22b7-4c7e-a3c0-75578515deb9","Type":"ContainerStarted","Data":"b4205acf6236434ac6e19178db2d5c88b76dce6514897f983bf61863fb5c9675"}
Nov 03 20:25:01.385625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:01.385013    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8" event={"ID":"39b8d333-b40a-4096-9de9-a944842f9544","Type":"ContainerStarted","Data":"8bb74a324f58b9f916e189f4656732c4ee7ba2a2fe2ed6c4522e459c15123f19"}
Nov 03 20:25:01.385625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:01.385083    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:25:01.385625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:01.385112    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8" event={"ID":"39b8d333-b40a-4096-9de9-a944842f9544","Type":"ContainerStarted","Data":"af1c12e2c885a64d22a2383253c22edfbd385442447513574bf83475f44306b6"}
Nov 03 20:25:01.428671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:01.428452    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87" podStartSLOduration=1.4284316590000001 podStartE2EDuration="1.428431659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:01.384859254 +0000 UTC m=+1431.517987729" watchObservedRunningTime="2026-11-03 20:25:01.428431659 +0000 UTC m=+1431.561560138"
Nov 03 20:25:01.432616 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:01.430450    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8" podStartSLOduration=1.43043645 podStartE2EDuration="1.43043645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:01.426778087 +0000 UTC m=+1431.559906544" watchObservedRunningTime="2026-11-03 20:25:01.43043645 +0000 UTC m=+1431.563564916"
Nov 03 20:25:01.694110 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:01.694078    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:01.825649 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:01.821700    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:01.926115 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:01.926079    4421 patch_prober.go:29] interesting pod/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 20:25:01.926296 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:01.926136    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5" podUID="8e6a0a0b-c9ca-4182-8962-fd4534a2844b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 20:25:02.699146 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:02.699108    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 03 20:25:02.699813 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:02.699780    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8" podUID="39b8d333-b40a-4096-9de9-a944842f9544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 03 20:25:02.825504 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:02.825474    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:25:02.825933 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:02.825902    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87" podUID="57010672-22b7-4c7e-a3c0-75578515deb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 20:25:02.926614 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:02.926576    4421 patch_prober.go:29] interesting pod/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 20:25:02.926796 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:02.926637    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5" podUID="8e6a0a0b-c9ca-4182-8962-fd4534a2844b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 20:25:03.706628 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:03.701011    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 03 20:25:03.706628 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:03.701076    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8" podUID="39b8d333-b40a-4096-9de9-a944842f9544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 03 20:25:03.826625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:03.823044    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:25:03.826625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:03.823122    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87" podUID="57010672-22b7-4c7e-a3c0-75578515deb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 20:25:03.926194 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:03.926149    4421 patch_prober.go:29] interesting pod/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 20:25:03.926418 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:03.926220    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5" podUID="8e6a0a0b-c9ca-4182-8962-fd4534a2844b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 20:25:04.703201 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:04.703146    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 03 20:25:04.703514 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:04.703467    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8" podUID="39b8d333-b40a-4096-9de9-a944842f9544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 03 20:25:04.823159 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:04.823123    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:25:04.823854 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:04.823803    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87" podUID="57010672-22b7-4c7e-a3c0-75578515deb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 20:25:04.957008 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:04.956922    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:25:04.965838 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:04.965810    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:25:05.697193 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:05.696714    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 03 20:25:05.697193 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:05.696770    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8" podUID="39b8d333-b40a-4096-9de9-a944842f9544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 03 20:25:05.823039 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:05.823003    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:25:05.823368 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:05.823339    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87" podUID="57010672-22b7-4c7e-a3c0-75578515deb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 20:25:06.700620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:06.697107    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 03 20:25:06.700620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:06.697182    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8" podUID="39b8d333-b40a-4096-9de9-a944842f9544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 03 20:25:06.828624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:06.825894    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:25:06.828624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:06.825953    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87" podUID="57010672-22b7-4c7e-a3c0-75578515deb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 20:25:07.693980 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:07.693936    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 03 20:25:07.694146 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:07.694023    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8" podUID="39b8d333-b40a-4096-9de9-a944842f9544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 03 20:25:07.767854 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:07.767821    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb"]
Nov 03 20:25:07.793170 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:07.793138    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl"]
Nov 03 20:25:07.823150 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:07.823118    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:25:07.823396 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:07.823371    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87" podUID="57010672-22b7-4c7e-a3c0-75578515deb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 20:25:08.694725 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:08.694681    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Nov 03 20:25:08.695205 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:08.694752    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8" podUID="39b8d333-b40a-4096-9de9-a944842f9544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Nov 03 20:25:08.821545 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:08.821504    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 20:25:08.821789 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:08.821577    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87" podUID="57010672-22b7-4c7e-a3c0-75578515deb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 20:25:09.698246 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:09.698217    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:09.698622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:09.698564    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:09.699232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:09.699209    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:25:09.825701 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:09.825677    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:09.826128 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:09.826105    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:09.826768 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:09.826752    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:25:10.547263 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.547232    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"]
Nov 03 20:25:10.551143 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.551117    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:10.575024 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.569487    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"]
Nov 03 20:25:10.724013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.723975    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:10.744730 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.744699    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f598b0-63c5-449a-b963-3d9921553aae-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:10.767622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.752723    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:10.767622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.762880    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:10.767622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.762952    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f598b0-63c5-449a-b963-3d9921553aae-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:10.767622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.763076    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2854\" (UniqueName: \"kubernetes.io/projected/61f598b0-63c5-449a-b963-3d9921553aae-kube-api-access-d2854\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:10.767622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.763184    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:10.767622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.763242    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f598b0-63c5-449a-b963-3d9921553aae-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:10.767622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.763344    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:10.865513 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.865420    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f598b0-63c5-449a-b963-3d9921553aae-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:10.866255 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.866228    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:10.866709 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.866688    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:10.869054 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.868991    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f598b0-63c5-449a-b963-3d9921553aae-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:10.869738 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.869717    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:10.870215 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.870197    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:10.870387 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.870372    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f598b0-63c5-449a-b963-3d9921553aae-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:10.870570 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.870552    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2854\" (UniqueName: \"kubernetes.io/projected/61f598b0-63c5-449a-b963-3d9921553aae-kube-api-access-d2854\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:10.870812 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.870795    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:10.871388 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.871362    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:10.871388 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.868183    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:10.871677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.871655    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:10.873107 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.872787    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f598b0-63c5-449a-b963-3d9921553aae-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:10.873451 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.868734    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:10.900922 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.900848    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:10.934570 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.934540    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f598b0-63c5-449a-b963-3d9921553aae-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:10.935761 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.935298    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f598b0-63c5-449a-b963-3d9921553aae-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:10.938540 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:10.938518    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2854\" (UniqueName: \"kubernetes.io/projected/61f598b0-63c5-449a-b963-3d9921553aae-kube-api-access-d2854\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:11.170965 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:11.170867    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:11.700134 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:11.688859    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"]
Nov 03 20:25:11.726089 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:11.726035    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:11.726517 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:11.726177    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:11.726517 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:11.726257    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:12.470622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:12.469412    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9"]
Nov 03 20:25:12.492677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:12.479753    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9" podUID="58d8fae7-6dba-491c-860e-05d2ec4e08f2" containerName="maas-api" containerID="cri-o://2046101b0a3e1af688613c54ecd7e4777a276e0e6dd1bdebb6b13feb2a15c2ac" gracePeriod=30
Nov 03 20:25:12.981620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:12.976325    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb" podUID="b1469175-be50-4567-9235-31dd037fd284" containerName="payload-processing" containerID="cri-o://6cdd6b373f8eecba75c960b67b36fa7b9aa7f2c8d4e4a2468d06837be36c9440" gracePeriod=25
Nov 03 20:25:13.031759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:12.992520    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl" podUID="cf8fcf01-c9dd-4a54-8d17-8986f7f27647" containerName="payload-pre-processing" containerID="cri-o://4081179fc2ae7ac39be6bddbde840faac96f13e327d7ea6bbf332bb753e9c966" gracePeriod=25
Nov 03 20:25:13.549623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:13.546053    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7" event={"ID":"61f598b0-63c5-449a-b963-3d9921553aae","Type":"ContainerStarted","Data":"d017ce5afee266d337a066c74c14643a2fedfdeac2a3d5b1b85406aed27511fb"}
Nov 03 20:25:13.549623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:13.546149    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7" event={"ID":"61f598b0-63c5-449a-b963-3d9921553aae","Type":"ContainerStarted","Data":"af4ff79520b876e3ced132e310d95ac8eb7f6bdd1eba282f9f93f16c96b185f7"}
Nov 03 20:25:13.600889 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:13.599200    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7" podStartSLOduration=3.599181171 podStartE2EDuration="3.599181171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:13.596355494 +0000 UTC m=+1443.729483956" watchObservedRunningTime="2026-11-03 20:25:13.599181171 +0000 UTC m=+1443.732309635"
Nov 03 20:25:13.999627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:13.994463    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:14.189719 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:14.188716    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:14.244643 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:14.243710    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 20:25:14.244643 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:14.243792    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7" podUID="61f598b0-63c5-449a-b963-3d9921553aae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 20:25:14.305634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:14.282468    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:14.722703 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:14.718711    4421 generic.go:356] "Generic (PLEG): container finished" podID="cf8fcf01-c9dd-4a54-8d17-8986f7f27647" containerID="4081179fc2ae7ac39be6bddbde840faac96f13e327d7ea6bbf332bb753e9c966" exitCode=0
Nov 03 20:25:14.722703 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:14.718913    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl" event={"ID":"cf8fcf01-c9dd-4a54-8d17-8986f7f27647","Type":"ContainerDied","Data":"4081179fc2ae7ac39be6bddbde840faac96f13e327d7ea6bbf332bb753e9c966"}
Nov 03 20:25:14.722703 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:14.721318    4421 generic.go:356] "Generic (PLEG): container finished" podID="58d8fae7-6dba-491c-860e-05d2ec4e08f2" containerID="2046101b0a3e1af688613c54ecd7e4777a276e0e6dd1bdebb6b13feb2a15c2ac" exitCode=0
Nov 03 20:25:14.722703 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:14.721438    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9" event={"ID":"58d8fae7-6dba-491c-860e-05d2ec4e08f2","Type":"ContainerDied","Data":"2046101b0a3e1af688613c54ecd7e4777a276e0e6dd1bdebb6b13feb2a15c2ac"}
Nov 03 20:25:14.729245 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:14.723948    4421 generic.go:356] "Generic (PLEG): container finished" podID="b1469175-be50-4567-9235-31dd037fd284" containerID="6cdd6b373f8eecba75c960b67b36fa7b9aa7f2c8d4e4a2468d06837be36c9440" exitCode=0
Nov 03 20:25:14.729245 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:14.724802    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb" event={"ID":"b1469175-be50-4567-9235-31dd037fd284","Type":"ContainerDied","Data":"6cdd6b373f8eecba75c960b67b36fa7b9aa7f2c8d4e4a2468d06837be36c9440"}
Nov 03 20:25:14.783463 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:25:14.783405    4421 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d8fae7_6dba_491c_860e_05d2ec4e08f2.slice/crio-2046101b0a3e1af688613c54ecd7e4777a276e0e6dd1bdebb6b13feb2a15c2ac.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d8fae7_6dba_491c_860e_05d2ec4e08f2.slice/crio-conmon-2046101b0a3e1af688613c54ecd7e4777a276e0e6dd1bdebb6b13feb2a15c2ac.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1469175_be50_4567_9235_31dd037fd284.slice/crio-conmon-6cdd6b373f8eecba75c960b67b36fa7b9aa7f2c8d4e4a2468d06837be36c9440.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf8fcf01_c9dd_4a54_8d17_8986f7f27647.slice/crio-conmon-4081179fc2ae7ac39be6bddbde840faac96f13e327d7ea6bbf332bb753e9c966.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf8fcf01_c9dd_4a54_8d17_8986f7f27647.slice/crio-4081179fc2ae7ac39be6bddbde840faac96f13e327d7ea6bbf332bb753e9c966.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:25:14.795013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:14.794982    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl"
Nov 03 20:25:14.917694 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:14.917553    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf8fcf01-c9dd-4a54-8d17-8986f7f27647-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8fcf01-c9dd-4a54-8d17-8986f7f27647-plugins-config-volume\") pod \"cf8fcf01-c9dd-4a54-8d17-8986f7f27647\" (UID: \"cf8fcf01-c9dd-4a54-8d17-8986f7f27647\") "
Nov 03 20:25:14.917906 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:14.917714    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf8fcf01-c9dd-4a54-8d17-8986f7f27647-kube-api-access-hbtgf\" (UniqueName: \"kubernetes.io/projected/cf8fcf01-c9dd-4a54-8d17-8986f7f27647-kube-api-access-hbtgf\") pod \"cf8fcf01-c9dd-4a54-8d17-8986f7f27647\" (UID: \"cf8fcf01-c9dd-4a54-8d17-8986f7f27647\") "
Nov 03 20:25:14.926441 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:14.918887    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf8fcf01-c9dd-4a54-8d17-8986f7f27647-plugins-config-volume" pod "cf8fcf01-c9dd-4a54-8d17-8986f7f27647" (UID: "cf8fcf01-c9dd-4a54-8d17-8986f7f27647"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:14.946630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:14.945220    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8fcf01-c9dd-4a54-8d17-8986f7f27647-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:15.028705 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.028657    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf8fcf01-c9dd-4a54-8d17-8986f7f27647-kube-api-access-hbtgf" pod "cf8fcf01-c9dd-4a54-8d17-8986f7f27647" (UID: "cf8fcf01-c9dd-4a54-8d17-8986f7f27647"). InnerVolumeSpecName "kube-api-access-hbtgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:15.056318 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.056269    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbtgf\" (UniqueName: \"kubernetes.io/projected/cf8fcf01-c9dd-4a54-8d17-8986f7f27647-kube-api-access-hbtgf\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:15.067695 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.067661    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9"
Nov 03 20:25:15.068814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.068793    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb"
Nov 03 20:25:15.158976 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.158944    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58d8fae7-6dba-491c-860e-05d2ec4e08f2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58d8fae7-6dba-491c-860e-05d2ec4e08f2-openshift-service-ca\") pod \"58d8fae7-6dba-491c-860e-05d2ec4e08f2\" (UID: \"58d8fae7-6dba-491c-860e-05d2ec4e08f2\") "
Nov 03 20:25:15.159280 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.159259    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58d8fae7-6dba-491c-860e-05d2ec4e08f2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58d8fae7-6dba-491c-860e-05d2ec4e08f2-maas-api-tls\") pod \"58d8fae7-6dba-491c-860e-05d2ec4e08f2\" (UID: \"58d8fae7-6dba-491c-860e-05d2ec4e08f2\") "
Nov 03 20:25:15.159460 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.159391    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1469175-be50-4567-9235-31dd037fd284-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1469175-be50-4567-9235-31dd037fd284-plugins-config-volume\") pod \"b1469175-be50-4567-9235-31dd037fd284\" (UID: \"b1469175-be50-4567-9235-31dd037fd284\") "
Nov 03 20:25:15.172699 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.167833    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58d8fae7-6dba-491c-860e-05d2ec4e08f2-kube-api-access-mf276\" (UniqueName: \"kubernetes.io/projected/58d8fae7-6dba-491c-860e-05d2ec4e08f2-kube-api-access-mf276\") pod \"58d8fae7-6dba-491c-860e-05d2ec4e08f2\" (UID: \"58d8fae7-6dba-491c-860e-05d2ec4e08f2\") "
Nov 03 20:25:15.172699 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.167939    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1469175-be50-4567-9235-31dd037fd284-kube-api-access-jqmzl\" (UniqueName: \"kubernetes.io/projected/b1469175-be50-4567-9235-31dd037fd284-kube-api-access-jqmzl\") pod \"b1469175-be50-4567-9235-31dd037fd284\" (UID: \"b1469175-be50-4567-9235-31dd037fd284\") "
Nov 03 20:25:15.172699 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.168012    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58d8fae7-6dba-491c-860e-05d2ec4e08f2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/58d8fae7-6dba-491c-860e-05d2ec4e08f2-metrics-certs\") pod \"58d8fae7-6dba-491c-860e-05d2ec4e08f2\" (UID: \"58d8fae7-6dba-491c-860e-05d2ec4e08f2\") "
Nov 03 20:25:15.172699 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.160281    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58d8fae7-6dba-491c-860e-05d2ec4e08f2-openshift-service-ca" pod "58d8fae7-6dba-491c-860e-05d2ec4e08f2" (UID: "58d8fae7-6dba-491c-860e-05d2ec4e08f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:15.172699 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.170852    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1469175-be50-4567-9235-31dd037fd284-plugins-config-volume" pod "b1469175-be50-4567-9235-31dd037fd284" (UID: "b1469175-be50-4567-9235-31dd037fd284"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:15.172699 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.172067    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 20:25:15.172699 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.172116    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7" podUID="61f598b0-63c5-449a-b963-3d9921553aae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 20:25:15.172699 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.172260    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58d8fae7-6dba-491c-860e-05d2ec4e08f2-metrics-certs" pod "58d8fae7-6dba-491c-860e-05d2ec4e08f2" (UID: "58d8fae7-6dba-491c-860e-05d2ec4e08f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:25:15.173860 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.173834    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58d8fae7-6dba-491c-860e-05d2ec4e08f2-maas-api-tls" pod "58d8fae7-6dba-491c-860e-05d2ec4e08f2" (UID: "58d8fae7-6dba-491c-860e-05d2ec4e08f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:25:15.174001 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.173872    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1469175-be50-4567-9235-31dd037fd284-kube-api-access-jqmzl" pod "b1469175-be50-4567-9235-31dd037fd284" (UID: "b1469175-be50-4567-9235-31dd037fd284"). InnerVolumeSpecName "kube-api-access-jqmzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:15.175554 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.175526    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d8fae7-6dba-491c-860e-05d2ec4e08f2-kube-api-access-mf276" pod "58d8fae7-6dba-491c-860e-05d2ec4e08f2" (UID: "58d8fae7-6dba-491c-860e-05d2ec4e08f2"). InnerVolumeSpecName "kube-api-access-mf276". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:15.291621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.285715    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58d8fae7-6dba-491c-860e-05d2ec4e08f2-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:15.291621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.285752    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1469175-be50-4567-9235-31dd037fd284-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:15.291621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.285778    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf276\" (UniqueName: \"kubernetes.io/projected/58d8fae7-6dba-491c-860e-05d2ec4e08f2-kube-api-access-mf276\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:15.291621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.285792    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqmzl\" (UniqueName: \"kubernetes.io/projected/b1469175-be50-4567-9235-31dd037fd284-kube-api-access-jqmzl\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:15.291621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.285805    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58d8fae7-6dba-491c-860e-05d2ec4e08f2-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:15.291621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.285818    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58d8fae7-6dba-491c-860e-05d2ec4e08f2-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:15.515169 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.515132    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"]
Nov 03 20:25:15.515420 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.515390    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5" podUID="8e6a0a0b-c9ca-4182-8962-fd4534a2844b" containerName="istio-proxy" containerID="cri-o://4f72288219f98fd4f341a1c01d4d0c0124b475d51d9a244a9c430be2715939f5" gracePeriod=30
Nov 03 20:25:15.763624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.763102    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb" event={"ID":"b1469175-be50-4567-9235-31dd037fd284","Type":"ContainerDied","Data":"beeb17d27d0f92fef578e2ce5615ef5c820cf8d3c28222fda7308d667ab2be07"}
Nov 03 20:25:15.763624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.763184    4421 scope.go:122] "RemoveContainer" containerID="6cdd6b373f8eecba75c960b67b36fa7b9aa7f2c8d4e4a2468d06837be36c9440"
Nov 03 20:25:15.763624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.763339    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb"
Nov 03 20:25:15.776933 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.776892    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl" event={"ID":"cf8fcf01-c9dd-4a54-8d17-8986f7f27647","Type":"ContainerDied","Data":"24ba4caaeac0ce91bdd95efe485762db6352faa0cb3bf4998e56140594241476"}
Nov 03 20:25:15.777266 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.777248    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl"
Nov 03 20:25:15.844662 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.844202    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9"
Nov 03 20:25:15.855638 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.849906    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9" event={"ID":"58d8fae7-6dba-491c-860e-05d2ec4e08f2","Type":"ContainerDied","Data":"1b1afeb51ef44979401d95ccdbee34966dd9477a31e8c5f1daead70f7b78fcb5"}
Nov 03 20:25:15.867624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.859887    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb"]
Nov 03 20:25:15.915147 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.915120    4421 scope.go:122] "RemoveContainer" containerID="4081179fc2ae7ac39be6bddbde840faac96f13e327d7ea6bbf332bb753e9c966"
Nov 03 20:25:15.918944 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.918209    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-59bc3b-6f849ff658-8gvkb"]
Nov 03 20:25:15.929176 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.929100    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9"]
Nov 03 20:25:15.935356 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.934785    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-59bc3b-9c8cb686f-rncg9"]
Nov 03 20:25:15.938886 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.938869    4421 scope.go:122] "RemoveContainer" containerID="2046101b0a3e1af688613c54ecd7e4777a276e0e6dd1bdebb6b13feb2a15c2ac"
Nov 03 20:25:15.939682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.939665    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl"]
Nov 03 20:25:15.980248 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:15.962028    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-59bc3b-58ccf8847f-kdtdl"]
Nov 03 20:25:16.016525 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:16.016495    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58d8fae7-6dba-491c-860e-05d2ec4e08f2" path="/var/lib/kubelet/pods/58d8fae7-6dba-491c-860e-05d2ec4e08f2/volumes"
Nov 03 20:25:16.017016 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:16.016999    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1469175-be50-4567-9235-31dd037fd284" path="/var/lib/kubelet/pods/b1469175-be50-4567-9235-31dd037fd284/volumes"
Nov 03 20:25:16.017410 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:16.017396    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf8fcf01-c9dd-4a54-8d17-8986f7f27647" path="/var/lib/kubelet/pods/cf8fcf01-c9dd-4a54-8d17-8986f7f27647/volumes"
Nov 03 20:25:16.172004 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:16.171922    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 20:25:16.172004 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:16.171974    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7" podUID="61f598b0-63c5-449a-b963-3d9921553aae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 20:25:17.171956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:17.171919    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 20:25:17.172142 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:17.171987    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7" podUID="61f598b0-63c5-449a-b963-3d9921553aae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 20:25:18.171826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:18.171794    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 20:25:18.172028 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:18.171867    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7" podUID="61f598b0-63c5-449a-b963-3d9921553aae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 20:25:19.172479 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:19.172438    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 20:25:19.173081 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:19.172508    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7" podUID="61f598b0-63c5-449a-b963-3d9921553aae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 20:25:19.635127 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:25:19.635073    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:25:19.960743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:19.960708    4421 patch_prober.go:29] interesting pod/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:25:19.960927 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:19.960785    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5" podUID="8e6a0a0b-c9ca-4182-8962-fd4534a2844b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:25:20.206407 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.206376    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:20.206972 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.206949    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:20.208232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.208208    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:25:20.818963 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.818941    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:25:20.903956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.903916    4421 generic.go:356] "Generic (PLEG): container finished" podID="8e6a0a0b-c9ca-4182-8962-fd4534a2844b" containerID="4f72288219f98fd4f341a1c01d4d0c0124b475d51d9a244a9c430be2715939f5" exitCode=0
Nov 03 20:25:20.904127 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.903982    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"
Nov 03 20:25:20.904127 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.903996    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5" event={"ID":"8e6a0a0b-c9ca-4182-8962-fd4534a2844b","Type":"ContainerDied","Data":"4f72288219f98fd4f341a1c01d4d0c0124b475d51d9a244a9c430be2715939f5"}
Nov 03 20:25:20.904127 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.904053    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5" event={"ID":"8e6a0a0b-c9ca-4182-8962-fd4534a2844b","Type":"ContainerDied","Data":"900f94ce3388fb55026f045963e51799c6eebfd8f86bfdd922c2fac9eb378b86"}
Nov 03 20:25:20.904127 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.904081    4421 scope.go:122] "RemoveContainer" containerID="4f72288219f98fd4f341a1c01d4d0c0124b475d51d9a244a9c430be2715939f5"
Nov 03 20:25:20.914636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.914591    4421 scope.go:122] "RemoveContainer" containerID="e563aea5ed0451dc85ae7b158bd943a4689594239571b31a5583cf922ea4f331"
Nov 03 20:25:20.923761 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.923740    4421 scope.go:122] "RemoveContainer" containerID="4f72288219f98fd4f341a1c01d4d0c0124b475d51d9a244a9c430be2715939f5"
Nov 03 20:25:20.924005 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:25:20.923989    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f72288219f98fd4f341a1c01d4d0c0124b475d51d9a244a9c430be2715939f5\": container with ID starting with 4f72288219f98fd4f341a1c01d4d0c0124b475d51d9a244a9c430be2715939f5 not found: ID does not exist" containerID="4f72288219f98fd4f341a1c01d4d0c0124b475d51d9a244a9c430be2715939f5"
Nov 03 20:25:20.924057 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.924014    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f72288219f98fd4f341a1c01d4d0c0124b475d51d9a244a9c430be2715939f5"} err="failed to get container status \"4f72288219f98fd4f341a1c01d4d0c0124b475d51d9a244a9c430be2715939f5\": rpc error: code = NotFound desc = could not find container \"4f72288219f98fd4f341a1c01d4d0c0124b475d51d9a244a9c430be2715939f5\": container with ID starting with 4f72288219f98fd4f341a1c01d4d0c0124b475d51d9a244a9c430be2715939f5 not found: ID does not exist"
Nov 03 20:25:20.924057 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.924031    4421 scope.go:122] "RemoveContainer" containerID="e563aea5ed0451dc85ae7b158bd943a4689594239571b31a5583cf922ea4f331"
Nov 03 20:25:20.924269 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:25:20.924253    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e563aea5ed0451dc85ae7b158bd943a4689594239571b31a5583cf922ea4f331\": container with ID starting with e563aea5ed0451dc85ae7b158bd943a4689594239571b31a5583cf922ea4f331 not found: ID does not exist" containerID="e563aea5ed0451dc85ae7b158bd943a4689594239571b31a5583cf922ea4f331"
Nov 03 20:25:20.924321 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.924274    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e563aea5ed0451dc85ae7b158bd943a4689594239571b31a5583cf922ea4f331"} err="failed to get container status \"e563aea5ed0451dc85ae7b158bd943a4689594239571b31a5583cf922ea4f331\": rpc error: code = NotFound desc = could not find container \"e563aea5ed0451dc85ae7b158bd943a4689594239571b31a5583cf922ea4f331\": container with ID starting with e563aea5ed0451dc85ae7b158bd943a4689594239571b31a5583cf922ea4f331 not found: ID does not exist"
Nov 03 20:25:20.926773 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.926754    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-workload-certs\") pod \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") "
Nov 03 20:25:20.926835 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.926799    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-token\" (UniqueName: \"kubernetes.io/projected/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-token\") pod \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") "
Nov 03 20:25:20.926869 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.926845    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-envoy\") pod \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") "
Nov 03 20:25:20.926911 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.926895    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-data\") pod \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") "
Nov 03 20:25:20.926969 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.926926    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-credential-socket\") pod \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") "
Nov 03 20:25:20.926969 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.926955    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-podinfo\") pod \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") "
Nov 03 20:25:20.927072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.926971    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-workload-certs" pod "8e6a0a0b-c9ca-4182-8962-fd4534a2844b" (UID: "8e6a0a0b-c9ca-4182-8962-fd4534a2844b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:20.927072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.927000    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-workload-socket\") pod \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") "
Nov 03 20:25:20.927172 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.927125    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istiod-ca-cert\") pod \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") "
Nov 03 20:25:20.927230 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.927180    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-kube-api-access-wvl55\" (UniqueName: \"kubernetes.io/projected/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-kube-api-access-wvl55\") pod \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\" (UID: \"8e6a0a0b-c9ca-4182-8962-fd4534a2844b\") "
Nov 03 20:25:20.927292 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.927232    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-credential-socket" pod "8e6a0a0b-c9ca-4182-8962-fd4534a2844b" (UID: "8e6a0a0b-c9ca-4182-8962-fd4534a2844b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:20.927348 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.927298    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-data" pod "8e6a0a0b-c9ca-4182-8962-fd4534a2844b" (UID: "8e6a0a0b-c9ca-4182-8962-fd4534a2844b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:20.927635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.927523    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istiod-ca-cert" pod "8e6a0a0b-c9ca-4182-8962-fd4534a2844b" (UID: "8e6a0a0b-c9ca-4182-8962-fd4534a2844b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:20.928148 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.928125    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:20.928234 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.928156    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:20.928234 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.928172    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:20.928234 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.928188    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:20.928462 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.928439    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-workload-socket" pod "8e6a0a0b-c9ca-4182-8962-fd4534a2844b" (UID: "8e6a0a0b-c9ca-4182-8962-fd4534a2844b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:20.929278 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.929244    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-token" pod "8e6a0a0b-c9ca-4182-8962-fd4534a2844b" (UID: "8e6a0a0b-c9ca-4182-8962-fd4534a2844b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:20.929278 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.929244    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-envoy" pod "8e6a0a0b-c9ca-4182-8962-fd4534a2844b" (UID: "8e6a0a0b-c9ca-4182-8962-fd4534a2844b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:25:20.929403 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.929351    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-podinfo" pod "8e6a0a0b-c9ca-4182-8962-fd4534a2844b" (UID: "8e6a0a0b-c9ca-4182-8962-fd4534a2844b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:25:20.929437 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:20.929417    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-kube-api-access-wvl55" pod "8e6a0a0b-c9ca-4182-8962-fd4534a2844b" (UID: "8e6a0a0b-c9ca-4182-8962-fd4534a2844b"). InnerVolumeSpecName "kube-api-access-wvl55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:21.029774 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:21.029741    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:21.029774 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:21.029773    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:21.029774 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:21.029783    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:21.030102 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:21.029793    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:21.030102 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:21.029801    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvl55\" (UniqueName: \"kubernetes.io/projected/8e6a0a0b-c9ca-4182-8962-fd4534a2844b-kube-api-access-wvl55\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:21.229430 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:21.229404    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"]
Nov 03 20:25:21.234177 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:21.234154    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-59bc3b-openshift-default-54cf78d8b5-gn2g5"]
Nov 03 20:25:22.018697 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:22.018665    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e6a0a0b-c9ca-4182-8962-fd4534a2844b" path="/var/lib/kubelet/pods/8e6a0a0b-c9ca-4182-8962-fd4534a2844b/volumes"
Nov 03 20:25:24.297928 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:24.297895    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2_7021016a-c1fd-4b0c-849b-7e8470447465/payload-processing/0.log"
Nov 03 20:25:24.490194 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:24.490170    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-xx9xb_79a63610-3e85-4ddd-a7a8-6ceccc2b3f83/payload-processing/0.log"
Nov 03 20:25:24.676348 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:24.676259    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-xx9xb_79a63610-3e85-4ddd-a7a8-6ceccc2b3f83/payload-processing/0.log"
Nov 03 20:25:24.853110 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:24.853083    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl_d3fd1d16-f04c-4807-8991-1987e5245fb8/payload-processing/0.log"
Nov 03 20:25:25.057335 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:25.057307    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl_d3fd1d16-f04c-4807-8991-1987e5245fb8/payload-processing/0.log"
Nov 03 20:25:34.628734 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:34.628685    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:34.783833 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:34.782628    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:34.981624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:34.969810    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:35.318682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:35.305959    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:35.369620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:35.354956    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 03 20:25:35.620123 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:35.619256    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:25:35.731550 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:35.720301    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:35.731550 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:35.721979    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:25:35.789627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:35.783885    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:35.789627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:35.784646    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:25:35.981787 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:35.975690    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:36.055681 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:36.025703    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:25:36.269679 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:36.269000    4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:36.310170 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:36.307907    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:36.327015 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:36.326916    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:25:36.327342 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:36.327286    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7" podUID="61f598b0-63c5-449a-b963-3d9921553aae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:36.374660 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:36.355309    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 03 20:25:36.571752 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:36.560393    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:25:36.571752 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:36.560466    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:25:36.640763 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:36.640717    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:36.676456 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:36.644590    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:25:36.698325 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:36.682704    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:25:36.698325 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:36.682763    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47" podUID="b270c5c6-0444-4410-a70d-37edb799adce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:36.959150 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:36.959105    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:25:37.010626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:36.985739    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:25:37.046320 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:36.983445    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 20:25:37.407686 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:37.374630    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:37.429662 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:37.376153    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:37.676683 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:37.651966    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:37.988761 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:37.988708    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:38.009752 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:38.009708    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:25:38.010049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:38.010016    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn" podUID="d31c0deb-63e5-42a2-a34c-90499e741b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:38.036628 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:38.024724    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:38.058165 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:38.058125    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 20:25:38.421680 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:38.420562    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:38.470756 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:38.470705    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:38.597723 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:38.597676    4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:38.772634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:38.771879    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:25:38.914875 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:38.914847    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:25:39.008225 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:38.991549    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:39.064720 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:39.064035    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:39.219796 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:39.219430    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:25:39.223060 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:39.223032    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:25:39.227546 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:39.227520    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:25:39.284637 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:39.284562    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:25:39.301341 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:39.301301    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:25:49.629034 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:25:49.628953    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:25:50.472947 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.472913    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"]
Nov 03 20:25:50.473531 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.473517    4421 cpu_manager.go:414] "containerMap: removing container" podUID="cf8fcf01-c9dd-4a54-8d17-8986f7f27647" containerName="payload-pre-processing"
Nov 03 20:25:50.473580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.473534    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf8fcf01-c9dd-4a54-8d17-8986f7f27647" containerName="payload-pre-processing"
Nov 03 20:25:50.473580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.473543    4421 cpu_manager.go:414] "containerMap: removing container" podUID="b1469175-be50-4567-9235-31dd037fd284" containerName="payload-processing"
Nov 03 20:25:50.473580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.473547    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1469175-be50-4567-9235-31dd037fd284" containerName="payload-processing"
Nov 03 20:25:50.473580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.473569    4421 cpu_manager.go:414] "containerMap: removing container" podUID="58d8fae7-6dba-491c-860e-05d2ec4e08f2" containerName="maas-api"
Nov 03 20:25:50.473580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.473574    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58d8fae7-6dba-491c-860e-05d2ec4e08f2" containerName="maas-api"
Nov 03 20:25:50.473790 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.473588    4421 cpu_manager.go:414] "containerMap: removing container" podUID="8e6a0a0b-c9ca-4182-8962-fd4534a2844b" containerName="istio-proxy"
Nov 03 20:25:50.473790 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.473607    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e6a0a0b-c9ca-4182-8962-fd4534a2844b" containerName="istio-proxy"
Nov 03 20:25:50.473790 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.473617    4421 cpu_manager.go:414] "containerMap: removing container" podUID="8e6a0a0b-c9ca-4182-8962-fd4534a2844b" containerName="istio-proxy"
Nov 03 20:25:50.473790 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.473621    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e6a0a0b-c9ca-4182-8962-fd4534a2844b" containerName="istio-proxy"
Nov 03 20:25:50.473790 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.473714    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e6a0a0b-c9ca-4182-8962-fd4534a2844b" containerName="istio-proxy"
Nov 03 20:25:50.473790 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.473727    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf8fcf01-c9dd-4a54-8d17-8986f7f27647" containerName="payload-pre-processing"
Nov 03 20:25:50.473790 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.473734    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1469175-be50-4567-9235-31dd037fd284" containerName="payload-processing"
Nov 03 20:25:50.473790 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.473742    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="58d8fae7-6dba-491c-860e-05d2ec4e08f2" containerName="maas-api"
Nov 03 20:25:50.478345 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.478325    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.490722 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.490695    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"]
Nov 03 20:25:50.502892 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.502863    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42p5g\" (UniqueName: \"kubernetes.io/projected/484992e4-0e7a-4728-9adb-0106b20d4dc0-kube-api-access-42p5g\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.503849 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.503820    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.504063 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.504036    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484992e4-0e7a-4728-9adb-0106b20d4dc0-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.504152 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.504081    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.504239 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.504224    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.504328 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.504314    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.504405 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.504390    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.504511 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.504485    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.504669 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.504647    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.606119 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.606086    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.606297 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.606128    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.606297 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.606265    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.606419 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.606394    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.606484 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.606433    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.606484 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.606454    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42p5g\" (UniqueName: \"kubernetes.io/projected/484992e4-0e7a-4728-9adb-0106b20d4dc0-kube-api-access-42p5g\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.606586 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.606533    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.606586 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.606551    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.606760 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.606631    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484992e4-0e7a-4728-9adb-0106b20d4dc0-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.606760 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.606666    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.606760 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.606726    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.606940 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.606791    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.607085 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.607055    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.607220 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.607204    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484992e4-0e7a-4728-9adb-0106b20d4dc0-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.608823 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.608795    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.609133 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.609116    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.682851 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.682816    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.683306 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.683206    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42p5g\" (UniqueName: \"kubernetes.io/projected/484992e4-0e7a-4728-9adb-0106b20d4dc0-kube-api-access-42p5g\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:50.796729 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:50.796659    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:51.084877 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:51.084849    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"]
Nov 03 20:25:51.085774 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:25:51.085740    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod484992e4_0e7a_4728_9adb_0106b20d4dc0.slice/crio-c591f062e3011f31bd21408dc8c475b542cacc0c2eeaa915c33657afd354815b WatchSource:0}: Error finding container c591f062e3011f31bd21408dc8c475b542cacc0c2eeaa915c33657afd354815b: Status 404 returned error can't find the container with id c591f062e3011f31bd21408dc8c475b542cacc0c2eeaa915c33657afd354815b
Nov 03 20:25:51.088429 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:51.088393    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:51.088512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:51.088461    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:51.088512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:51.088489    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:51.305733 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:51.305689    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx" event={"ID":"484992e4-0e7a-4728-9adb-0106b20d4dc0","Type":"ContainerStarted","Data":"156e81e44712c985e6dde2291cecf41fc8ca94fe9917ea0df651460cfac77979"}
Nov 03 20:25:51.305946 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:51.305772    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx" event={"ID":"484992e4-0e7a-4728-9adb-0106b20d4dc0","Type":"ContainerStarted","Data":"c591f062e3011f31bd21408dc8c475b542cacc0c2eeaa915c33657afd354815b"}
Nov 03 20:25:51.333929 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:51.333811    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx" podStartSLOduration=1.333796965 podStartE2EDuration="1.333796965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:51.331284416 +0000 UTC m=+1481.464412880" watchObservedRunningTime="2026-11-03 20:25:51.333796965 +0000 UTC m=+1481.466925428"
Nov 03 20:25:51.797066 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:51.797034    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:25:52.799679 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:52.799644    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 03 20:25:52.800179 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:52.799700    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx" podUID="484992e4-0e7a-4728-9adb-0106b20d4dc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 03 20:25:53.797945 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:53.797904    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 03 20:25:53.798117 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:53.797975    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx" podUID="484992e4-0e7a-4728-9adb-0106b20d4dc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 03 20:25:54.798841 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:54.798805    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 03 20:25:54.799419 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:54.799388    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx" podUID="484992e4-0e7a-4728-9adb-0106b20d4dc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 03 20:25:55.296623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.292229    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"]
Nov 03 20:25:55.296623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.293346    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e6a0a0b-c9ca-4182-8962-fd4534a2844b" containerName="istio-proxy"
Nov 03 20:25:55.313619 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.308918    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:55.456744 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.374237    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-7a92d4-openshift-default-dockercfg-k7n7g\""
Nov 03 20:25:55.456744 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.374289    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"]
Nov 03 20:25:55.456744 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.376031    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-istio-envoy\") pod \"e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") " pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:55.456744 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.376069    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-credential-socket\") pod \"e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") " pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:55.456744 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.376126    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-workload-certs\") pod \"e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") " pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:55.456744 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.376155    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-istio-data\") pod \"e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") " pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:55.456744 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.376181    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f55c349-f323-4dde-baab-4e279bfcf422-istio-token\") pod \"e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") " pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:55.456744 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.376224    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-workload-socket\") pod \"e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") " pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:55.456744 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.376330    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5xtz\" (UniqueName: \"kubernetes.io/projected/4f55c349-f323-4dde-baab-4e279bfcf422-kube-api-access-s5xtz\") pod \"e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") " pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:55.456744 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.376365    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f55c349-f323-4dde-baab-4e279bfcf422-istio-podinfo\") pod \"e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") " pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:55.456744 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.376514    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f55c349-f323-4dde-baab-4e279bfcf422-istiod-ca-cert\") pod \"e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") " pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:55.625325 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.623291    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f55c349-f323-4dde-baab-4e279bfcf422-istiod-ca-cert\") pod \"e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") " pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:55.625325 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.623427    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-istio-envoy\") pod \"e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") " pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:55.625325 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.623458    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-credential-socket\") pod \"e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") " pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:55.625325 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.623515    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-workload-certs\") pod \"e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") " pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:55.625325 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.623542    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-istio-data\") pod \"e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") " pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:55.625325 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.623564    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f55c349-f323-4dde-baab-4e279bfcf422-istio-token\") pod \"e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") " pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:55.625325 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.623632    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-workload-socket\") pod \"e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") " pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:55.625325 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.623758    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5xtz\" (UniqueName: \"kubernetes.io/projected/4f55c349-f323-4dde-baab-4e279bfcf422-kube-api-access-s5xtz\") pod \"e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") " pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:55.625325 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.623797    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f55c349-f323-4dde-baab-4e279bfcf422-istio-podinfo\") pod \"e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") " pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:55.625325 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.624927    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f55c349-f323-4dde-baab-4e279bfcf422-istiod-ca-cert\") pod \"e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") " pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:55.630336 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.630306    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-workload-socket\") pod \"e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") " pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:55.654680 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.630877    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-credential-socket\") pod \"e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") " pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:55.655295 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.655269    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-workload-certs\") pod \"e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") " pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:55.666656 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.655705    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-istio-data\") pod \"e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") " pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:55.795433 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.795399    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f55c349-f323-4dde-baab-4e279bfcf422-istio-token\") pod \"e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") " pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:55.821620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.821569    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f55c349-f323-4dde-baab-4e279bfcf422-istio-podinfo\") pod \"e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") " pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:55.924689 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.918144    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 03 20:25:55.924689 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.918208    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx" podUID="484992e4-0e7a-4728-9adb-0106b20d4dc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 03 20:25:55.945741 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.944103    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5xtz\" (UniqueName: \"kubernetes.io/projected/4f55c349-f323-4dde-baab-4e279bfcf422-kube-api-access-s5xtz\") pod \"e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") " pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:56.011625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:55.990869    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:56.082629 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:56.071679    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:56.226627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:56.219755    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:25:56.478738 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:56.478155    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:56.521626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:56.505697    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-istio-envoy\") pod \"e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") " pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:56.538624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:56.531126    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:25:56.770797 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:56.770701    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:56.798313 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:56.798274    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 03 20:25:56.798570 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:56.798538    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx" podUID="484992e4-0e7a-4728-9adb-0106b20d4dc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 03 20:25:57.031802 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:57.031696    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:25:57.054631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:57.048809    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:57.055482 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:57.055455    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:25:57.094651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:57.074572    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:57.094651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:57.089277    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:25:57.223738 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:57.222814    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:25:57.225676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:57.225654    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:25:57.481626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:57.478379    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:57.481626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:57.480086    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:25:57.841108 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:57.841003    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:57.923625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:57.843239    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 03 20:25:57.923625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:57.920274    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx" podUID="484992e4-0e7a-4728-9adb-0106b20d4dc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 03 20:25:57.923625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:57.920728    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:25:58.066626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:58.041718    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:58.144476 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:58.132168    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:58.144476 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:58.133273    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:25:58.234212 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:58.234164    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:25:58.547764 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:58.547717    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 03 20:25:58.887386 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:58.749209    4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:58.887646 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:58.768311    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:25:58.888482 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:58.834385    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 03 20:25:58.892700 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:58.892664    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx" podUID="484992e4-0e7a-4728-9adb-0106b20d4dc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 03 20:25:58.916022 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:58.915988    4421 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:25:58.916262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:58.916230    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw" podUID="317e6b68-68f8-4220-a3ef-2f0999a19ad9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:58.935361 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:58.935319    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:59.071626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:59.068213    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:59.181675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:59.148679    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:59.181675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:59.148764    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:59.529720 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:59.367826    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:59.668740 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:59.668686    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:59.675624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:59.670721    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:59.708634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:59.676800    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:25:59.867028 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:59.844968    4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded"
Nov 03 20:25:59.943620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:59.928429    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:25:59.972470 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:25:59.971915    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:00.158255 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:00.158153    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 20:26:00.182234 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:00.182187    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 03 20:26:00.182441 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:00.182244    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx" podUID="484992e4-0e7a-4728-9adb-0106b20d4dc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 03 20:26:00.183323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:00.182565    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:00.371378 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:00.371335    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:00.762822 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:00.762788    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:26:00.763038 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:00.763018    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb" event={"ID":"4f55c349-f323-4dde-baab-4e279bfcf422","Type":"ContainerStarted","Data":"4b3c3506f83ddd4b9cd5b07519515fcc77e47dd973994249bd9ac44a4501722c"}
Nov 03 20:26:00.763265 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:00.763252    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:26:00.802172 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:00.802138    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 03 20:26:00.802395 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:00.802373    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx" podUID="484992e4-0e7a-4728-9adb-0106b20d4dc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 03 20:26:00.816122 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:00.815451    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:26:00.889827 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:00.880821    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:26:00.908397 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:00.908222    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"]
Nov 03 20:26:00.908809 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:00.908775    4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:00.915897 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:00.915870    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"]
Nov 03 20:26:00.953617 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:00.952852    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"]
Nov 03 20:26:00.953617 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:00.953070    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.062724 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.057362    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:26:01.062724 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.061580    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:26:01.066298 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.066247    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.066449 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.066339    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.066449 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.066392    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/989c3294-e402-4c4d-8369-ca96d89fa5cd-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.066449 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.066422    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.066675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.066473    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.066675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.066520    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.066797 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.066712    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.066797 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.066784    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gl7g\" (UniqueName: \"kubernetes.io/projected/989c3294-e402-4c4d-8369-ca96d89fa5cd-kube-api-access-2gl7g\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.066907 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.066828    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.200078 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.200012    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gl7g\" (UniqueName: \"kubernetes.io/projected/989c3294-e402-4c4d-8369-ca96d89fa5cd-kube-api-access-2gl7g\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.200818 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.200794    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.201407 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.201349    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.201736 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.201718    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.201994 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.201975    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/989c3294-e402-4c4d-8369-ca96d89fa5cd-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.202167 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.202150    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.202380 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.202364    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.202570 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.202554    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.203113 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.203080    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.205267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.205238    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.241694 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.206274    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.241694 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.206517    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.241694 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.229885    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/989c3294-e402-4c4d-8369-ca96d89fa5cd-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.241694 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.234693    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.247744 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.245647    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.249221 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.249109    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.251829 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.243726    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gl7g\" (UniqueName: \"kubernetes.io/projected/989c3294-e402-4c4d-8369-ca96d89fa5cd-kube-api-access-2gl7g\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.254248 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.254196    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.394308 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.394217    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:01.632350 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.632314    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb" event={"ID":"4f55c349-f323-4dde-baab-4e279bfcf422","Type":"ContainerStarted","Data":"71a6959fc201244bd7cd969d7edf1cc4c88495e6e053bb2e90440313b33923cf"}
Nov 03 20:26:01.658571 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.658478    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb" podStartSLOduration=6.6584650960000005 podStartE2EDuration="6.658465096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:01.654747381 +0000 UTC m=+1491.787875868" watchObservedRunningTime="2026-11-03 20:26:01.658465096 +0000 UTC m=+1491.791593617"
Nov 03 20:26:01.693493 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.693467    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"]
Nov 03 20:26:01.694782 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:26:01.694754    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989c3294_e402_4c4d_8369_ca96d89fa5cd.slice/crio-81f904131f3e83befbd19954bff673900f4b3ced173bbae407c783d38165fe1a WatchSource:0}: Error finding container 81f904131f3e83befbd19954bff673900f4b3ced173bbae407c783d38165fe1a: Status 404 returned error can't find the container with id 81f904131f3e83befbd19954bff673900f4b3ced173bbae407c783d38165fe1a
Nov 03 20:26:01.697849 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.697815    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:01.697931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.697882    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:01.697931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.697914    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:01.822787 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.817310    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Nov 03 20:26:01.822787 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:01.817366    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx" podUID="484992e4-0e7a-4728-9adb-0106b20d4dc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Nov 03 20:26:02.546806 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:02.546772    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:26:02.561982 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:02.561951    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:26:02.664623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:02.663758    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5" event={"ID":"989c3294-e402-4c4d-8369-ca96d89fa5cd","Type":"ContainerStarted","Data":"21eac988b1a95ccb158ef275484c9b99b715b60fbb79d72d5c59e8004e70ce71"}
Nov 03 20:26:02.664623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:02.663918    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5" event={"ID":"989c3294-e402-4c4d-8369-ca96d89fa5cd","Type":"ContainerStarted","Data":"81f904131f3e83befbd19954bff673900f4b3ced173bbae407c783d38165fe1a"}
Nov 03 20:26:02.668621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:02.664922    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:26:02.672988 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:02.672950    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:26:02.696812 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:02.696763    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5" podStartSLOduration=3.696747955 podStartE2EDuration="3.696747955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:02.693811057 +0000 UTC m=+1492.826939522" watchObservedRunningTime="2026-11-03 20:26:02.696747955 +0000 UTC m=+1492.829876492"
Nov 03 20:26:03.395191 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:03.395148    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:03.402621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:03.401561    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 03 20:26:03.402621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:03.401640    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5" podUID="989c3294-e402-4c4d-8369-ca96d89fa5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 03 20:26:03.839739 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:03.839708    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:26:03.840219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:03.839759    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx" podUID="484992e4-0e7a-4728-9adb-0106b20d4dc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:04.178067 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:04.177057    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:26:04.182706 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:04.178760    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:26:04.187620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:04.185908    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:26:04.398225 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:04.395098    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 03 20:26:04.398225 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:04.395144    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5" podUID="989c3294-e402-4c4d-8369-ca96d89fa5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 03 20:26:05.118088 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:05.118056    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb"]
Nov 03 20:26:05.122817 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:05.122792    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb"
Nov 03 20:26:05.126990 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:05.126961    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc"]
Nov 03 20:26:05.130735 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:05.130695    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-7a92d4\""
Nov 03 20:26:05.139928 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:05.139176    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-7a92d4-dockercfg-fqp5s\""
Nov 03 20:26:05.159208 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:05.155791    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb"]
Nov 03 20:26:05.159208 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:05.155945    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc"
Nov 03 20:26:05.203155 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:05.193683    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc"]
Nov 03 20:26:05.258644 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:05.257464    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkhq7\" (UniqueName: \"kubernetes.io/projected/c70b0ee2-5d82-4575-86ee-a7adc291b6ff-kube-api-access-jkhq7\") pod \"payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb\" (UID: \"c70b0ee2-5d82-4575-86ee-a7adc291b6ff\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb"
Nov 03 20:26:05.258644 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:05.257644    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c70b0ee2-5d82-4575-86ee-a7adc291b6ff-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb\" (UID: \"c70b0ee2-5d82-4575-86ee-a7adc291b6ff\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb"
Nov 03 20:26:05.370674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:05.364719    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjmxm\" (UniqueName: \"kubernetes.io/projected/0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab-kube-api-access-gjmxm\") pod \"payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc\" (UID: \"0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc"
Nov 03 20:26:05.370674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:05.364899    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc\" (UID: \"0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc"
Nov 03 20:26:05.370674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:05.364974    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkhq7\" (UniqueName: \"kubernetes.io/projected/c70b0ee2-5d82-4575-86ee-a7adc291b6ff-kube-api-access-jkhq7\") pod \"payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb\" (UID: \"c70b0ee2-5d82-4575-86ee-a7adc291b6ff\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb"
Nov 03 20:26:05.370674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:05.365125    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c70b0ee2-5d82-4575-86ee-a7adc291b6ff-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb\" (UID: \"c70b0ee2-5d82-4575-86ee-a7adc291b6ff\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb"
Nov 03 20:26:05.370674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:05.366065    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c70b0ee2-5d82-4575-86ee-a7adc291b6ff-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb\" (UID: \"c70b0ee2-5d82-4575-86ee-a7adc291b6ff\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb"
Nov 03 20:26:05.465622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:05.464981    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 03 20:26:05.465622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:05.465041    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5" podUID="989c3294-e402-4c4d-8369-ca96d89fa5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 03 20:26:05.483130 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:05.483065    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc\" (UID: \"0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc"
Nov 03 20:26:05.495152 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:05.495114    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjmxm\" (UniqueName: \"kubernetes.io/projected/0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab-kube-api-access-gjmxm\") pod \"payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc\" (UID: \"0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc"
Nov 03 20:26:05.519173 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:05.509199    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc\" (UID: \"0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc"
Nov 03 20:26:05.519173 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:05.516108    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkhq7\" (UniqueName: \"kubernetes.io/projected/c70b0ee2-5d82-4575-86ee-a7adc291b6ff-kube-api-access-jkhq7\") pod \"payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb\" (UID: \"c70b0ee2-5d82-4575-86ee-a7adc291b6ff\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb"
Nov 03 20:26:05.579346 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:05.579231    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjmxm\" (UniqueName: \"kubernetes.io/projected/0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab-kube-api-access-gjmxm\") pod \"payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc\" (UID: \"0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc"
Nov 03 20:26:05.797492 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:05.797461    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb"
Nov 03 20:26:05.805026 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:05.804999    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc"
Nov 03 20:26:06.288075 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:06.288045    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc"]
Nov 03 20:26:06.293207 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:26:06.293179    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a987b5a_d8ae_43ff_8bf2_b9cfa56916ab.slice/crio-d3f0c2e236bc8b5e92c72caa7096338e26a60a7309c3786b1ebf2f37bd230004 WatchSource:0}: Error finding container d3f0c2e236bc8b5e92c72caa7096338e26a60a7309c3786b1ebf2f37bd230004: Status 404 returned error can't find the container with id d3f0c2e236bc8b5e92c72caa7096338e26a60a7309c3786b1ebf2f37bd230004
Nov 03 20:26:06.293344 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:06.293326    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb"]
Nov 03 20:26:06.294094 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:26:06.294054    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc70b0ee2_5d82_4575_86ee_a7adc291b6ff.slice/crio-e6ce31a2f936666647ef7e999024a613bbfcda3430878278426328755ab0dd30 WatchSource:0}: Error finding container e6ce31a2f936666647ef7e999024a613bbfcda3430878278426328755ab0dd30: Status 404 returned error can't find the container with id e6ce31a2f936666647ef7e999024a613bbfcda3430878278426328755ab0dd30
Nov 03 20:26:06.394536 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:06.394511    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 03 20:26:06.394658 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:06.394548    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5" podUID="989c3294-e402-4c4d-8369-ca96d89fa5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 03 20:26:06.787403 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:06.787231    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb" event={"ID":"c70b0ee2-5d82-4575-86ee-a7adc291b6ff","Type":"ContainerStarted","Data":"5f6621dfeee05b538fe82a1c85388fd9525c729f11f906950f14e417232feea5"}
Nov 03 20:26:06.787403 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:06.787339    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb"
Nov 03 20:26:06.787403 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:06.787365    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb" event={"ID":"c70b0ee2-5d82-4575-86ee-a7adc291b6ff","Type":"ContainerStarted","Data":"e6ce31a2f936666647ef7e999024a613bbfcda3430878278426328755ab0dd30"}
Nov 03 20:26:06.789838 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:06.789813    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc" event={"ID":"0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab","Type":"ContainerStarted","Data":"b8f37ef2fa2157a640af667dbf722552dab3602b37e5e3b58b6e5814ae53dba7"}
Nov 03 20:26:06.790031 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:06.790008    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc" event={"ID":"0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab","Type":"ContainerStarted","Data":"d3f0c2e236bc8b5e92c72caa7096338e26a60a7309c3786b1ebf2f37bd230004"}
Nov 03 20:26:06.790140 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:06.790038    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc"
Nov 03 20:26:06.812713 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:06.812673    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb" podStartSLOduration=1.812660231 podStartE2EDuration="1.812660231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:06.80834234 +0000 UTC m=+1496.941470803" watchObservedRunningTime="2026-11-03 20:26:06.812660231 +0000 UTC m=+1496.945788694"
Nov 03 20:26:06.827061 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:06.827010    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc" podStartSLOduration=1.8269974310000001 podStartE2EDuration="1.826997431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:06.826218874 +0000 UTC m=+1496.959347337" watchObservedRunningTime="2026-11-03 20:26:06.826997431 +0000 UTC m=+1496.960125893"
Nov 03 20:26:07.394871 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:07.394833    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 03 20:26:07.395263 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:07.394892    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5" podUID="989c3294-e402-4c4d-8369-ca96d89fa5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 03 20:26:08.395268 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:08.395235    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 03 20:26:08.395691 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:08.395295    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5" podUID="989c3294-e402-4c4d-8369-ca96d89fa5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 03 20:26:09.848333 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:09.848305    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:09.849106 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:09.849078    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:09.850061 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:09.850042    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:26:10.709792 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:10.709738    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq"]
Nov 03 20:26:10.714564 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:10.714545    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq"
Nov 03 20:26:10.727678 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:10.722975    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-7a92d4-serving-cert\""
Nov 03 20:26:10.744113 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:10.743973    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq"]
Nov 03 20:26:10.890072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:10.888347    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33274f28-f2d3-40fc-a16b-75e90ece007a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq\" (UID: \"33274f28-f2d3-40fc-a16b-75e90ece007a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq"
Nov 03 20:26:10.899299 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:10.891975    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33274f28-f2d3-40fc-a16b-75e90ece007a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq\" (UID: \"33274f28-f2d3-40fc-a16b-75e90ece007a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq"
Nov 03 20:26:10.899299 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:10.892184    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33274f28-f2d3-40fc-a16b-75e90ece007a-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq\" (UID: \"33274f28-f2d3-40fc-a16b-75e90ece007a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq"
Nov 03 20:26:10.899299 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:10.892291    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hnwv\" (UniqueName: \"kubernetes.io/projected/33274f28-f2d3-40fc-a16b-75e90ece007a-kube-api-access-5hnwv\") pod \"maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq\" (UID: \"33274f28-f2d3-40fc-a16b-75e90ece007a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq"
Nov 03 20:26:10.993829 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:10.993725    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33274f28-f2d3-40fc-a16b-75e90ece007a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq\" (UID: \"33274f28-f2d3-40fc-a16b-75e90ece007a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq"
Nov 03 20:26:10.994002 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:10.993884    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33274f28-f2d3-40fc-a16b-75e90ece007a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq\" (UID: \"33274f28-f2d3-40fc-a16b-75e90ece007a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq"
Nov 03 20:26:10.994002 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:10.993972    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33274f28-f2d3-40fc-a16b-75e90ece007a-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq\" (UID: \"33274f28-f2d3-40fc-a16b-75e90ece007a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq"
Nov 03 20:26:10.994002 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:26:10.993997    4421 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-serving-cert: secret "maas-api-e2e-shared-a-w1-7a92d4-serving-cert" not found
Nov 03 20:26:10.994197 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:26:10.994064    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33274f28-f2d3-40fc-a16b-75e90ece007a-maas-api-tls podName:33274f28-f2d3-40fc-a16b-75e90ece007a nodeName:}" failed. No retries permitted until 2026-11-03 20:26:11.494042356 +0000 UTC m=+1501.627170800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33274f28-f2d3-40fc-a16b-75e90ece007a-maas-api-tls") pod "maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq" (UID: "33274f28-f2d3-40fc-a16b-75e90ece007a") : secret "maas-api-e2e-shared-a-w1-7a92d4-serving-cert" not found
Nov 03 20:26:10.994197 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:10.994094    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hnwv\" (UniqueName: \"kubernetes.io/projected/33274f28-f2d3-40fc-a16b-75e90ece007a-kube-api-access-5hnwv\") pod \"maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq\" (UID: \"33274f28-f2d3-40fc-a16b-75e90ece007a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq"
Nov 03 20:26:10.994740 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:10.994719    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33274f28-f2d3-40fc-a16b-75e90ece007a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq\" (UID: \"33274f28-f2d3-40fc-a16b-75e90ece007a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq"
Nov 03 20:26:10.997102 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:10.997076    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33274f28-f2d3-40fc-a16b-75e90ece007a-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq\" (UID: \"33274f28-f2d3-40fc-a16b-75e90ece007a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq"
Nov 03 20:26:11.011912 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:11.011891    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hnwv\" (UniqueName: \"kubernetes.io/projected/33274f28-f2d3-40fc-a16b-75e90ece007a-kube-api-access-5hnwv\") pod \"maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq\" (UID: \"33274f28-f2d3-40fc-a16b-75e90ece007a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq"
Nov 03 20:26:11.542291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:11.542244    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33274f28-f2d3-40fc-a16b-75e90ece007a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq\" (UID: \"33274f28-f2d3-40fc-a16b-75e90ece007a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq"
Nov 03 20:26:11.545105 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:11.545076    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33274f28-f2d3-40fc-a16b-75e90ece007a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq\" (UID: \"33274f28-f2d3-40fc-a16b-75e90ece007a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq"
Nov 03 20:26:11.645323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:11.645292    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq"
Nov 03 20:26:12.186777 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:12.186753    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq"]
Nov 03 20:26:12.188656 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:26:12.188587    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33274f28_f2d3_40fc_a16b_75e90ece007a.slice/crio-39cde2908d2e15fd7d24343dbf88f208eb81f0787889403c04e5ce439c4564db WatchSource:0}: Error finding container 39cde2908d2e15fd7d24343dbf88f208eb81f0787889403c04e5ce439c4564db: Status 404 returned error can't find the container with id 39cde2908d2e15fd7d24343dbf88f208eb81f0787889403c04e5ce439c4564db
Nov 03 20:26:13.023620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:13.023389    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq" event={"ID":"33274f28-f2d3-40fc-a16b-75e90ece007a","Type":"ContainerStarted","Data":"56227fd565e459ced9ed09f25db9fd23e1542adbf202fd10c647f90c1566c64f"}
Nov 03 20:26:13.023620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:13.023458    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq" event={"ID":"33274f28-f2d3-40fc-a16b-75e90ece007a","Type":"ContainerStarted","Data":"39cde2908d2e15fd7d24343dbf88f208eb81f0787889403c04e5ce439c4564db"}
Nov 03 20:26:13.049621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:13.043751    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq"
Nov 03 20:26:14.938508 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:14.938460    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:14.976622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:14.962550    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 03 20:26:15.141622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:15.140019    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:15.141622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:15.140424    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:15.241368 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:15.238691    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:15.368934 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:15.368891    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:15.966631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:15.950786    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:15.966631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:15.951279    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:26:15.966631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:15.963112    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:15.979648 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:15.974867    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:26:16.175624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:16.166989    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:16.184948 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:16.183668    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 20:26:16.276681 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:16.251941    4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:16.407620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:16.256922    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:26:16.407620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:16.380692    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:16.690618 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:16.687050    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq" podStartSLOduration=6.20152425 podStartE2EDuration="6.687028297s" totalImagesPullingTime="485.504047ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:13.095006915 +0000 UTC m=+1503.228135379" watchObservedRunningTime="2026-11-03 20:26:16.687028297 +0000 UTC m=+1506.820156773"
Nov 03 20:26:16.866632 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:16.821983    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:26:16.866990 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:16.866966    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:26:16.867192 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:16.867177    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:26:16.887371 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:16.887340    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:26:16.996949 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:16.996848    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:17.026833 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:17.026791    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:17.229903 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:17.213919    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:17.231319 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:17.230206    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:26:17.373281 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:17.373179    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:17.393808 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:17.393757    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:17.398770 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:17.382276    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:26:17.557284 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:17.556735    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:26:17.886362 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:17.886332    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc"
Nov 03 20:26:17.886755 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:17.886736    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb"
Nov 03 20:26:18.037625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:18.037564    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:18.347627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:18.326061    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:18.402768 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:18.402718    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:18.430856 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:18.403046    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:18.653524 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:18.653440    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:26:18.653524 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:18.653495    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:26:18.653902 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:18.653533    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:26:18.653902 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:18.653561    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:26:19.630905 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:26:19.630859    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:26:20.216160 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:20.216132    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq"
Nov 03 20:26:24.887214 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:24.887177    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz"]
Nov 03 20:26:24.892651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:24.892627    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f"]
Nov 03 20:26:24.892811 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:24.892752    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz"
Nov 03 20:26:24.897015 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:24.896996    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f"
Nov 03 20:26:24.902282 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:24.902260    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz"]
Nov 03 20:26:24.906580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:24.906560    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-3c25bf-dockercfg-66glb\""
Nov 03 20:26:24.907035 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:24.906928    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-3c25bf\""
Nov 03 20:26:24.909319 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:24.909292    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f"]
Nov 03 20:26:24.989160 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:24.989126    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5fdfb8c-95f4-47c8-8154-4864c5485f24-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f\" (UID: \"e5fdfb8c-95f4-47c8-8154-4864c5485f24\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f"
Nov 03 20:26:24.992565 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:24.992532    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5cfn\" (UniqueName: \"kubernetes.io/projected/8fdce1ab-1499-40bf-8f67-f1f23b2959b8-kube-api-access-s5cfn\") pod \"payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz\" (UID: \"8fdce1ab-1499-40bf-8f67-f1f23b2959b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz"
Nov 03 20:26:24.993153 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:24.993129    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fdce1ab-1499-40bf-8f67-f1f23b2959b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz\" (UID: \"8fdce1ab-1499-40bf-8f67-f1f23b2959b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz"
Nov 03 20:26:24.993381 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:24.993363    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ckjd\" (UniqueName: \"kubernetes.io/projected/e5fdfb8c-95f4-47c8-8154-4864c5485f24-kube-api-access-5ckjd\") pod \"payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f\" (UID: \"e5fdfb8c-95f4-47c8-8154-4864c5485f24\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f"
Nov 03 20:26:25.095692 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:25.095652    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5cfn\" (UniqueName: \"kubernetes.io/projected/8fdce1ab-1499-40bf-8f67-f1f23b2959b8-kube-api-access-s5cfn\") pod \"payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz\" (UID: \"8fdce1ab-1499-40bf-8f67-f1f23b2959b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz"
Nov 03 20:26:25.096147 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:25.096120    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fdce1ab-1499-40bf-8f67-f1f23b2959b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz\" (UID: \"8fdce1ab-1499-40bf-8f67-f1f23b2959b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz"
Nov 03 20:26:25.096314 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:25.096297    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ckjd\" (UniqueName: \"kubernetes.io/projected/e5fdfb8c-95f4-47c8-8154-4864c5485f24-kube-api-access-5ckjd\") pod \"payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f\" (UID: \"e5fdfb8c-95f4-47c8-8154-4864c5485f24\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f"
Nov 03 20:26:25.101908 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:25.096588    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5fdfb8c-95f4-47c8-8154-4864c5485f24-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f\" (UID: \"e5fdfb8c-95f4-47c8-8154-4864c5485f24\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f"
Nov 03 20:26:25.106615 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:25.103145    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5fdfb8c-95f4-47c8-8154-4864c5485f24-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f\" (UID: \"e5fdfb8c-95f4-47c8-8154-4864c5485f24\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f"
Nov 03 20:26:25.109442 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:25.109401    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fdce1ab-1499-40bf-8f67-f1f23b2959b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz\" (UID: \"8fdce1ab-1499-40bf-8f67-f1f23b2959b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz"
Nov 03 20:26:25.140436 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:25.140359    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5cfn\" (UniqueName: \"kubernetes.io/projected/8fdce1ab-1499-40bf-8f67-f1f23b2959b8-kube-api-access-s5cfn\") pod \"payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz\" (UID: \"8fdce1ab-1499-40bf-8f67-f1f23b2959b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz"
Nov 03 20:26:25.144702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:25.143242    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ckjd\" (UniqueName: \"kubernetes.io/projected/e5fdfb8c-95f4-47c8-8154-4864c5485f24-kube-api-access-5ckjd\") pod \"payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f\" (UID: \"e5fdfb8c-95f4-47c8-8154-4864c5485f24\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f"
Nov 03 20:26:25.219664 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:25.218162    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz"
Nov 03 20:26:25.253631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:25.243421    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6"]
Nov 03 20:26:25.323055 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:25.323031    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2"]
Nov 03 20:26:25.323380 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:25.321873    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f"
Nov 03 20:26:25.339904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:25.323389    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6" podUID="8d0951ba-32cd-4e85-842a-2b487b838320" containerName="maas-api" containerID="cri-o://3185ddfbe8b11ceb64cb1f42b3892f4d5445206c70ed6850fc50336fe87854aa" gracePeriod=30
Nov 03 20:26:25.339904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:25.323681    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2" podUID="7021016a-c1fd-4b0c-849b-7e8470447465" containerName="payload-processing" containerID="cri-o://b92644618bd397b3b46d8fc5680aa5dda0b84cc015a58eadadd33bada7a675e9" gracePeriod=30
Nov 03 20:26:25.930382 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:25.930353    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz"]
Nov 03 20:26:25.940551 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:25.940062    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f"]
Nov 03 20:26:25.950155 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:25.950127    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6"
Nov 03 20:26:26.014078 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.013193    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2"
Nov 03 20:26:26.117659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.114866    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d0951ba-32cd-4e85-842a-2b487b838320-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d0951ba-32cd-4e85-842a-2b487b838320-metrics-certs\") pod \"8d0951ba-32cd-4e85-842a-2b487b838320\" (UID: \"8d0951ba-32cd-4e85-842a-2b487b838320\") "
Nov 03 20:26:26.117659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.114966    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7021016a-c1fd-4b0c-849b-7e8470447465-kube-api-access-2q6hk\" (UniqueName: \"kubernetes.io/projected/7021016a-c1fd-4b0c-849b-7e8470447465-kube-api-access-2q6hk\") pod \"7021016a-c1fd-4b0c-849b-7e8470447465\" (UID: \"7021016a-c1fd-4b0c-849b-7e8470447465\") "
Nov 03 20:26:26.117659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.115172    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d0951ba-32cd-4e85-842a-2b487b838320-kube-api-access-jwtkr\" (UniqueName: \"kubernetes.io/projected/8d0951ba-32cd-4e85-842a-2b487b838320-kube-api-access-jwtkr\") pod \"8d0951ba-32cd-4e85-842a-2b487b838320\" (UID: \"8d0951ba-32cd-4e85-842a-2b487b838320\") "
Nov 03 20:26:26.117659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.115272    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d0951ba-32cd-4e85-842a-2b487b838320-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d0951ba-32cd-4e85-842a-2b487b838320-openshift-service-ca\") pod \"8d0951ba-32cd-4e85-842a-2b487b838320\" (UID: \"8d0951ba-32cd-4e85-842a-2b487b838320\") "
Nov 03 20:26:26.117659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.115340    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7021016a-c1fd-4b0c-849b-7e8470447465-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7021016a-c1fd-4b0c-849b-7e8470447465-plugins-config-volume\") pod \"7021016a-c1fd-4b0c-849b-7e8470447465\" (UID: \"7021016a-c1fd-4b0c-849b-7e8470447465\") "
Nov 03 20:26:26.117659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.115377    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d0951ba-32cd-4e85-842a-2b487b838320-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d0951ba-32cd-4e85-842a-2b487b838320-maas-api-tls\") pod \"8d0951ba-32cd-4e85-842a-2b487b838320\" (UID: \"8d0951ba-32cd-4e85-842a-2b487b838320\") "
Nov 03 20:26:26.170580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.160796    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7021016a-c1fd-4b0c-849b-7e8470447465-kube-api-access-2q6hk" pod "7021016a-c1fd-4b0c-849b-7e8470447465" (UID: "7021016a-c1fd-4b0c-849b-7e8470447465"). InnerVolumeSpecName "kube-api-access-2q6hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:26.170580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.165829    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d0951ba-32cd-4e85-842a-2b487b838320-metrics-certs" pod "8d0951ba-32cd-4e85-842a-2b487b838320" (UID: "8d0951ba-32cd-4e85-842a-2b487b838320"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:26.170580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.166219    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d0951ba-32cd-4e85-842a-2b487b838320-openshift-service-ca" pod "8d0951ba-32cd-4e85-842a-2b487b838320" (UID: "8d0951ba-32cd-4e85-842a-2b487b838320"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:26.170580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.166387    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7021016a-c1fd-4b0c-849b-7e8470447465-plugins-config-volume" pod "7021016a-c1fd-4b0c-849b-7e8470447465" (UID: "7021016a-c1fd-4b0c-849b-7e8470447465"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:26.170580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.169144    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d0951ba-32cd-4e85-842a-2b487b838320-maas-api-tls" pod "8d0951ba-32cd-4e85-842a-2b487b838320" (UID: "8d0951ba-32cd-4e85-842a-2b487b838320"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:26.170580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.169984    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q6hk\" (UniqueName: \"kubernetes.io/projected/7021016a-c1fd-4b0c-849b-7e8470447465-kube-api-access-2q6hk\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.170580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.170011    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d0951ba-32cd-4e85-842a-2b487b838320-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.170580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.170025    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7021016a-c1fd-4b0c-849b-7e8470447465-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.170580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.170039    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d0951ba-32cd-4e85-842a-2b487b838320-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.170580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.170054    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d0951ba-32cd-4e85-842a-2b487b838320-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.182290 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.182140    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d0951ba-32cd-4e85-842a-2b487b838320-kube-api-access-jwtkr" pod "8d0951ba-32cd-4e85-842a-2b487b838320" (UID: "8d0951ba-32cd-4e85-842a-2b487b838320"). InnerVolumeSpecName "kube-api-access-jwtkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:26.271864 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.271834    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwtkr\" (UniqueName: \"kubernetes.io/projected/8d0951ba-32cd-4e85-842a-2b487b838320-kube-api-access-jwtkr\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:26.426554 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.426465    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz" event={"ID":"8fdce1ab-1499-40bf-8f67-f1f23b2959b8","Type":"ContainerStarted","Data":"58e383371f8740126034176d160b5f6654e2d22c13387627dd5e07bdbebe4d9f"}
Nov 03 20:26:26.426554 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.426534    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz" event={"ID":"8fdce1ab-1499-40bf-8f67-f1f23b2959b8","Type":"ContainerStarted","Data":"5a67df43959f9888bb2a92224c786a7b96ab5f09080f348c6e043bea63abe098"}
Nov 03 20:26:26.426792 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.426568    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz"
Nov 03 20:26:26.428309 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.428275    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f" event={"ID":"e5fdfb8c-95f4-47c8-8154-4864c5485f24","Type":"ContainerStarted","Data":"321e25ca960e84c979fba6dce522d466156d2dfba9d597d60e50437f0e40c0ab"}
Nov 03 20:26:26.428465 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.428321    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f" event={"ID":"e5fdfb8c-95f4-47c8-8154-4864c5485f24","Type":"ContainerStarted","Data":"8459761267dafad76199c0a959bb08cb6d0e4b8c04edd1a3d7c9d63e997243a1"}
Nov 03 20:26:26.428465 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.428346    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f"
Nov 03 20:26:26.429630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.429590    4421 generic.go:356] "Generic (PLEG): container finished" podID="7021016a-c1fd-4b0c-849b-7e8470447465" containerID="b92644618bd397b3b46d8fc5680aa5dda0b84cc015a58eadadd33bada7a675e9" exitCode=0
Nov 03 20:26:26.429718 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.429639    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2"
Nov 03 20:26:26.429718 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.429669    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2" event={"ID":"7021016a-c1fd-4b0c-849b-7e8470447465","Type":"ContainerDied","Data":"b92644618bd397b3b46d8fc5680aa5dda0b84cc015a58eadadd33bada7a675e9"}
Nov 03 20:26:26.429718 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.429710    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2" event={"ID":"7021016a-c1fd-4b0c-849b-7e8470447465","Type":"ContainerDied","Data":"79665edeb2717b3492bfc2bd0874022fdfc8bc2df511bffb58863c4e7c7fb667"}
Nov 03 20:26:26.429890 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.429738    4421 scope.go:122] "RemoveContainer" containerID="b92644618bd397b3b46d8fc5680aa5dda0b84cc015a58eadadd33bada7a675e9"
Nov 03 20:26:26.431098 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.431084    4421 generic.go:356] "Generic (PLEG): container finished" podID="8d0951ba-32cd-4e85-842a-2b487b838320" containerID="3185ddfbe8b11ceb64cb1f42b3892f4d5445206c70ed6850fc50336fe87854aa" exitCode=0
Nov 03 20:26:26.431155 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.431146    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6" event={"ID":"8d0951ba-32cd-4e85-842a-2b487b838320","Type":"ContainerDied","Data":"3185ddfbe8b11ceb64cb1f42b3892f4d5445206c70ed6850fc50336fe87854aa"}
Nov 03 20:26:26.431228 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.431171    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6" event={"ID":"8d0951ba-32cd-4e85-842a-2b487b838320","Type":"ContainerDied","Data":"5b7b564c9367244c17fcb35818f30129d3c3ab872aa7a8650cc279f1007e51d6"}
Nov 03 20:26:26.431228 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.431222    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6"
Nov 03 20:26:26.441428 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.441409    4421 scope.go:122] "RemoveContainer" containerID="b92644618bd397b3b46d8fc5680aa5dda0b84cc015a58eadadd33bada7a675e9"
Nov 03 20:26:26.441756 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:26:26.441728    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b92644618bd397b3b46d8fc5680aa5dda0b84cc015a58eadadd33bada7a675e9\": container with ID starting with b92644618bd397b3b46d8fc5680aa5dda0b84cc015a58eadadd33bada7a675e9 not found: ID does not exist" containerID="b92644618bd397b3b46d8fc5680aa5dda0b84cc015a58eadadd33bada7a675e9"
Nov 03 20:26:26.441872 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.441820    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b92644618bd397b3b46d8fc5680aa5dda0b84cc015a58eadadd33bada7a675e9"} err="failed to get container status \"b92644618bd397b3b46d8fc5680aa5dda0b84cc015a58eadadd33bada7a675e9\": rpc error: code = NotFound desc = could not find container \"b92644618bd397b3b46d8fc5680aa5dda0b84cc015a58eadadd33bada7a675e9\": container with ID starting with b92644618bd397b3b46d8fc5680aa5dda0b84cc015a58eadadd33bada7a675e9 not found: ID does not exist"
Nov 03 20:26:26.441872 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.441840    4421 scope.go:122] "RemoveContainer" containerID="3185ddfbe8b11ceb64cb1f42b3892f4d5445206c70ed6850fc50336fe87854aa"
Nov 03 20:26:26.446873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.446832    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz" podStartSLOduration=2.446819411 podStartE2EDuration="2.446819411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:26.444553707 +0000 UTC m=+1516.577682167" watchObservedRunningTime="2026-11-03 20:26:26.446819411 +0000 UTC m=+1516.579947872"
Nov 03 20:26:26.455039 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.455020    4421 scope.go:122] "RemoveContainer" containerID="3185ddfbe8b11ceb64cb1f42b3892f4d5445206c70ed6850fc50336fe87854aa"
Nov 03 20:26:26.455355 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:26:26.455330    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3185ddfbe8b11ceb64cb1f42b3892f4d5445206c70ed6850fc50336fe87854aa\": container with ID starting with 3185ddfbe8b11ceb64cb1f42b3892f4d5445206c70ed6850fc50336fe87854aa not found: ID does not exist" containerID="3185ddfbe8b11ceb64cb1f42b3892f4d5445206c70ed6850fc50336fe87854aa"
Nov 03 20:26:26.455422 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.455366    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3185ddfbe8b11ceb64cb1f42b3892f4d5445206c70ed6850fc50336fe87854aa"} err="failed to get container status \"3185ddfbe8b11ceb64cb1f42b3892f4d5445206c70ed6850fc50336fe87854aa\": rpc error: code = NotFound desc = could not find container \"3185ddfbe8b11ceb64cb1f42b3892f4d5445206c70ed6850fc50336fe87854aa\": container with ID starting with 3185ddfbe8b11ceb64cb1f42b3892f4d5445206c70ed6850fc50336fe87854aa not found: ID does not exist"
Nov 03 20:26:26.464773 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.464752    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6"]
Nov 03 20:26:26.470798 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.470776    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2eee8c-6d4cdddb98-hkvn6"]
Nov 03 20:26:26.481318 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.481279    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2"]
Nov 03 20:26:26.485388 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.485369    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2eee8c-5f66985b86-868h2"]
Nov 03 20:26:26.500239 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:26.500196    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f" podStartSLOduration=2.500184672 podStartE2EDuration="2.500184672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:26.498724172 +0000 UTC m=+1516.631852637" watchObservedRunningTime="2026-11-03 20:26:26.500184672 +0000 UTC m=+1516.633313134"
Nov 03 20:26:28.017518 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:28.017487    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7021016a-c1fd-4b0c-849b-7e8470447465" path="/var/lib/kubelet/pods/7021016a-c1fd-4b0c-849b-7e8470447465/volumes"
Nov 03 20:26:28.018059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:28.018041    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d0951ba-32cd-4e85-842a-2b487b838320" path="/var/lib/kubelet/pods/8d0951ba-32cd-4e85-842a-2b487b838320/volumes"
Nov 03 20:26:28.454716 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:28.454683    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"]
Nov 03 20:26:28.454946 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:28.454916    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq" podUID="f9ec3c59-274d-4af5-a710-751cf5367d3c" containerName="istio-proxy" containerID="cri-o://25dabfd5874956d61f410ca631a998a8b81437b121d42e494c4e9b3d46e70abc" gracePeriod=30
Nov 03 20:26:29.532886 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.532829    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"]
Nov 03 20:26:29.533947 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.533888    4421 cpu_manager.go:414] "containerMap: removing container" podUID="8d0951ba-32cd-4e85-842a-2b487b838320" containerName="maas-api"
Nov 03 20:26:29.533947 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.533910    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d0951ba-32cd-4e85-842a-2b487b838320" containerName="maas-api"
Nov 03 20:26:29.533947 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.533950    4421 cpu_manager.go:414] "containerMap: removing container" podUID="7021016a-c1fd-4b0c-849b-7e8470447465" containerName="payload-processing"
Nov 03 20:26:29.534323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.533959    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7021016a-c1fd-4b0c-849b-7e8470447465" containerName="payload-processing"
Nov 03 20:26:29.534323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.534104    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="7021016a-c1fd-4b0c-849b-7e8470447465" containerName="payload-processing"
Nov 03 20:26:29.534323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.534125    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d0951ba-32cd-4e85-842a-2b487b838320" containerName="maas-api"
Nov 03 20:26:29.538894 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.538872    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.540851 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.540824    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"]
Nov 03 20:26:29.543639 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.543588    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-ba89e0-openshift-default-dockercfg-rvpjm\""
Nov 03 20:26:29.649114 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.649080    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-podinfo\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.649114 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.649117    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-credential-socket\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.649339 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.649136    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-data\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.649339 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.649288    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-workload-socket\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.649339 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.649321    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-envoy\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.649500 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.649355    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-token\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.649557 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.649494    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65c5b\" (UniqueName: \"kubernetes.io/projected/d366ab06-4688-4c4b-b71b-66bb5d20167d-kube-api-access-65c5b\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.649662 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.649645    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d366ab06-4688-4c4b-b71b-66bb5d20167d-istiod-ca-cert\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.649820 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.649805    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-workload-certs\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.771621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.753831    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d366ab06-4688-4c4b-b71b-66bb5d20167d-istiod-ca-cert\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.771621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.754009    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-workload-certs\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.771621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.754150    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-podinfo\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.771621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.754180    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-credential-socket\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.771621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.754207    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-data\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.771621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.754273    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-workload-socket\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.771621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.754297    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-envoy\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.771621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.754336    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-token\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.771621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.754388    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65c5b\" (UniqueName: \"kubernetes.io/projected/d366ab06-4688-4c4b-b71b-66bb5d20167d-kube-api-access-65c5b\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.771621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.755525    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d366ab06-4688-4c4b-b71b-66bb5d20167d-istiod-ca-cert\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.771621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.755829    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-workload-certs\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.771621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.770568    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-podinfo\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.771621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.770879    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-credential-socket\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.771621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.771110    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-data\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.771621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.771352    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-workload-socket\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.802669 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.800713    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-envoy\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.819714 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.814508    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65c5b\" (UniqueName: \"kubernetes.io/projected/d366ab06-4688-4c4b-b71b-66bb5d20167d-kube-api-access-65c5b\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.825622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.822292    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-token\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:29.857619 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:29.857072    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:30.218040 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.218008    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz"]
Nov 03 20:26:30.223489 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.223469    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz"
Nov 03 20:26:30.228729 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.228703    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-08a7bc\""
Nov 03 20:26:30.236858 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.236830    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz"]
Nov 03 20:26:30.242465 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.242428    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"]
Nov 03 20:26:30.248398 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.248371    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj"]
Nov 03 20:26:30.249985 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:26:30.249960    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd366ab06_4688_4c4b_b71b_66bb5d20167d.slice/crio-dfc29380fdaccc30d3dccef5e170b527a7843d353a876f058976ce941fa84e40 WatchSource:0}: Error finding container dfc29380fdaccc30d3dccef5e170b527a7843d353a876f058976ce941fa84e40: Status 404 returned error can't find the container with id dfc29380fdaccc30d3dccef5e170b527a7843d353a876f058976ce941fa84e40
Nov 03 20:26:30.253477 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.253447    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:30.253572 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.253513    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:30.253729 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.253584    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:30.255937 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.255918    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj"
Nov 03 20:26:30.258066 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.258041    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj"]
Nov 03 20:26:30.276588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.276566    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-08a7bc-dockercfg-54xcc\""
Nov 03 20:26:30.389779 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.380991    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b3e4444-3d44-4623-94a7-d4748d93a6e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj\" (UID: \"3b3e4444-3d44-4623-94a7-d4748d93a6e7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj"
Nov 03 20:26:30.389779 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.381120    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqglr\" (UniqueName: \"kubernetes.io/projected/2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8-kube-api-access-nqglr\") pod \"payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz\" (UID: \"2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz"
Nov 03 20:26:30.389779 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.381160    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz\" (UID: \"2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz"
Nov 03 20:26:30.389779 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.381394    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gnfs\" (UniqueName: \"kubernetes.io/projected/3b3e4444-3d44-4623-94a7-d4748d93a6e7-kube-api-access-6gnfs\") pod \"payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj\" (UID: \"3b3e4444-3d44-4623-94a7-d4748d93a6e7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj"
Nov 03 20:26:30.511677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.506243    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqglr\" (UniqueName: \"kubernetes.io/projected/2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8-kube-api-access-nqglr\") pod \"payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz\" (UID: \"2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz"
Nov 03 20:26:30.566849 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.546648    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz\" (UID: \"2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz"
Nov 03 20:26:30.566849 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.546969    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gnfs\" (UniqueName: \"kubernetes.io/projected/3b3e4444-3d44-4623-94a7-d4748d93a6e7-kube-api-access-6gnfs\") pod \"payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj\" (UID: \"3b3e4444-3d44-4623-94a7-d4748d93a6e7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj"
Nov 03 20:26:30.566849 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.547103    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b3e4444-3d44-4623-94a7-d4748d93a6e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj\" (UID: \"3b3e4444-3d44-4623-94a7-d4748d93a6e7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj"
Nov 03 20:26:30.566849 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.548113    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b3e4444-3d44-4623-94a7-d4748d93a6e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj\" (UID: \"3b3e4444-3d44-4623-94a7-d4748d93a6e7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj"
Nov 03 20:26:30.566849 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.558015    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz\" (UID: \"2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz"
Nov 03 20:26:30.579622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.578977    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqglr\" (UniqueName: \"kubernetes.io/projected/2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8-kube-api-access-nqglr\") pod \"payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz\" (UID: \"2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz"
Nov 03 20:26:30.607181 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.607149    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gnfs\" (UniqueName: \"kubernetes.io/projected/3b3e4444-3d44-4623-94a7-d4748d93a6e7-kube-api-access-6gnfs\") pod \"payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj\" (UID: \"3b3e4444-3d44-4623-94a7-d4748d93a6e7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj"
Nov 03 20:26:30.621351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.621315    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs" event={"ID":"d366ab06-4688-4c4b-b71b-66bb5d20167d","Type":"ContainerStarted","Data":"c6b1e4c79dc0eeb303bea0d33ed7fd995f7f616229ef4c287e6e4f7c5fe371d6"}
Nov 03 20:26:30.621519 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.621380    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs" event={"ID":"d366ab06-4688-4c4b-b71b-66bb5d20167d","Type":"ContainerStarted","Data":"dfc29380fdaccc30d3dccef5e170b527a7843d353a876f058976ce941fa84e40"}
Nov 03 20:26:30.684226 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.684167    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs" podStartSLOduration=1.684149053 podStartE2EDuration="1.684149053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:30.67887733 +0000 UTC m=+1520.812005793" watchObservedRunningTime="2026-11-03 20:26:30.684149053 +0000 UTC m=+1520.817277520"
Nov 03 20:26:30.864675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.859641    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz"
Nov 03 20:26:30.864913 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.864697    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:30.883794 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:30.883753    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj"
Nov 03 20:26:31.185952 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:31.185923    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz"]
Nov 03 20:26:31.186392 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:26:31.186369    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f5f33b9_e3df_43c6_bf68_345c3d6ae3b8.slice/crio-e48ee52fbd406fc429aa26d7203c9c692e0359d6fabb5699ea98ed46fbcbf2ec WatchSource:0}: Error finding container e48ee52fbd406fc429aa26d7203c9c692e0359d6fabb5699ea98ed46fbcbf2ec: Status 404 returned error can't find the container with id e48ee52fbd406fc429aa26d7203c9c692e0359d6fabb5699ea98ed46fbcbf2ec
Nov 03 20:26:31.199446 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:31.199413    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj"]
Nov 03 20:26:31.203143 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:26:31.203118    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b3e4444_3d44_4623_94a7_d4748d93a6e7.slice/crio-b319357306b812fca8cf738e9c2fff5b5dc81b056b42aff3741fdf0ea2399d38 WatchSource:0}: Error finding container b319357306b812fca8cf738e9c2fff5b5dc81b056b42aff3741fdf0ea2399d38: Status 404 returned error can't find the container with id b319357306b812fca8cf738e9c2fff5b5dc81b056b42aff3741fdf0ea2399d38
Nov 03 20:26:31.630114 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:31.630073    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz" event={"ID":"2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8","Type":"ContainerStarted","Data":"9234a64384bc59f6b6ea26851255c46133d9ed9612573ab109e7f849caf61af4"}
Nov 03 20:26:31.630571 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:31.630185    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz" event={"ID":"2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8","Type":"ContainerStarted","Data":"e48ee52fbd406fc429aa26d7203c9c692e0359d6fabb5699ea98ed46fbcbf2ec"}
Nov 03 20:26:31.630571 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:31.630221    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz"
Nov 03 20:26:31.631858 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:31.631832    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj" event={"ID":"3b3e4444-3d44-4623-94a7-d4748d93a6e7","Type":"ContainerStarted","Data":"739f25c7d5f02607fa6c77b3829634de38ae9c9c2db3030db17af97718a7ddf8"}
Nov 03 20:26:31.631976 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:31.631873    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj" event={"ID":"3b3e4444-3d44-4623-94a7-d4748d93a6e7","Type":"ContainerStarted","Data":"b319357306b812fca8cf738e9c2fff5b5dc81b056b42aff3741fdf0ea2399d38"}
Nov 03 20:26:31.631976 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:31.631902    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj"
Nov 03 20:26:31.652946 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:31.652904    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz" podStartSLOduration=1.652891188 podStartE2EDuration="1.652891188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:31.648421808 +0000 UTC m=+1521.781550269" watchObservedRunningTime="2026-11-03 20:26:31.652891188 +0000 UTC m=+1521.786019650"
Nov 03 20:26:31.669524 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:31.669480    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj" podStartSLOduration=1.6694638670000002 podStartE2EDuration="1.669463867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:31.668445485 +0000 UTC m=+1521.801573949" watchObservedRunningTime="2026-11-03 20:26:31.669463867 +0000 UTC m=+1521.802592329"
Nov 03 20:26:31.872401 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:31.872365    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:32.206243 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:32.206209    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p"]
Nov 03 20:26:32.211161 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:32.211127    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p"
Nov 03 20:26:32.227401 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:32.226873    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-3c25bf-serving-cert\""
Nov 03 20:26:32.235683 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:32.234988    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p"]
Nov 03 20:26:32.304288 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:32.304248    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0130f392-da22-43d5-87e5-6f69d6475d7d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p\" (UID: \"0130f392-da22-43d5-87e5-6f69d6475d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p"
Nov 03 20:26:32.304525 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:32.304428    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98dps\" (UniqueName: \"kubernetes.io/projected/0130f392-da22-43d5-87e5-6f69d6475d7d-kube-api-access-98dps\") pod \"maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p\" (UID: \"0130f392-da22-43d5-87e5-6f69d6475d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p"
Nov 03 20:26:32.304650 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:32.304628    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0130f392-da22-43d5-87e5-6f69d6475d7d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p\" (UID: \"0130f392-da22-43d5-87e5-6f69d6475d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p"
Nov 03 20:26:32.304711 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:32.304671    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0130f392-da22-43d5-87e5-6f69d6475d7d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p\" (UID: \"0130f392-da22-43d5-87e5-6f69d6475d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p"
Nov 03 20:26:32.426621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:32.406139    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0130f392-da22-43d5-87e5-6f69d6475d7d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p\" (UID: \"0130f392-da22-43d5-87e5-6f69d6475d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p"
Nov 03 20:26:32.426913 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:32.426883    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0130f392-da22-43d5-87e5-6f69d6475d7d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p\" (UID: \"0130f392-da22-43d5-87e5-6f69d6475d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p"
Nov 03 20:26:32.427384 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:32.427354    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0130f392-da22-43d5-87e5-6f69d6475d7d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p\" (UID: \"0130f392-da22-43d5-87e5-6f69d6475d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p"
Nov 03 20:26:32.427503 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:32.427487    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98dps\" (UniqueName: \"kubernetes.io/projected/0130f392-da22-43d5-87e5-6f69d6475d7d-kube-api-access-98dps\") pod \"maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p\" (UID: \"0130f392-da22-43d5-87e5-6f69d6475d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p"
Nov 03 20:26:32.433818 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:32.433789    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0130f392-da22-43d5-87e5-6f69d6475d7d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p\" (UID: \"0130f392-da22-43d5-87e5-6f69d6475d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p"
Nov 03 20:26:32.435657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:32.435572    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0130f392-da22-43d5-87e5-6f69d6475d7d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p\" (UID: \"0130f392-da22-43d5-87e5-6f69d6475d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p"
Nov 03 20:26:32.454305 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:32.454274    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0130f392-da22-43d5-87e5-6f69d6475d7d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p\" (UID: \"0130f392-da22-43d5-87e5-6f69d6475d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p"
Nov 03 20:26:32.469726 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:32.462978    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98dps\" (UniqueName: \"kubernetes.io/projected/0130f392-da22-43d5-87e5-6f69d6475d7d-kube-api-access-98dps\") pod \"maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p\" (UID: \"0130f392-da22-43d5-87e5-6f69d6475d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p"
Nov 03 20:26:32.555135 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:32.555104    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p"
Nov 03 20:26:32.643462 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:32.643434    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:32.644635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:32.644571    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:26:32.686868 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:32.686685    4421 patch_prober.go:29] interesting pod/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:26:32.686868 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:32.686743    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq" podUID="f9ec3c59-274d-4af5-a710-751cf5367d3c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:26:32.855003 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:32.851537    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p"]
Nov 03 20:26:32.860147 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:26:32.860113    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0130f392_da22_43d5_87e5_6f69d6475d7d.slice/crio-665889268f8ca06bb46bc46e80ceeb5e31d08c7a79b5cfb7f4c0ab9a6288c2e1 WatchSource:0}: Error finding container 665889268f8ca06bb46bc46e80ceeb5e31d08c7a79b5cfb7f4c0ab9a6288c2e1: Status 404 returned error can't find the container with id 665889268f8ca06bb46bc46e80ceeb5e31d08c7a79b5cfb7f4c0ab9a6288c2e1
Nov 03 20:26:33.662626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:33.661526    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p" event={"ID":"0130f392-da22-43d5-87e5-6f69d6475d7d","Type":"ContainerStarted","Data":"fd51a771f71f1b01a6da015c7bb9123e4acc4cbc411bb681488e1d1fc1cb6449"}
Nov 03 20:26:33.662626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:33.661615    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p" event={"ID":"0130f392-da22-43d5-87e5-6f69d6475d7d","Type":"ContainerStarted","Data":"665889268f8ca06bb46bc46e80ceeb5e31d08c7a79b5cfb7f4c0ab9a6288c2e1"}
Nov 03 20:26:33.663459 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:33.663392    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p"
Nov 03 20:26:33.668491 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:33.668466    4421 generic.go:356] "Generic (PLEG): container finished" podID="f9ec3c59-274d-4af5-a710-751cf5367d3c" containerID="25dabfd5874956d61f410ca631a998a8b81437b121d42e494c4e9b3d46e70abc" exitCode=0
Nov 03 20:26:33.669788 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:33.669757    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq" event={"ID":"f9ec3c59-274d-4af5-a710-751cf5367d3c","Type":"ContainerDied","Data":"25dabfd5874956d61f410ca631a998a8b81437b121d42e494c4e9b3d46e70abc"}
Nov 03 20:26:33.669980 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:33.669962    4421 scope.go:122] "RemoveContainer" containerID="b4cd17d481610745a638bc486c86fee13831ff6e632d37b3f23195c9e777f271"
Nov 03 20:26:33.693537 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:33.693479    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p" podStartSLOduration=1.184332141 podStartE2EDuration="1.693462714s" totalImagesPullingTime="509.130573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:33.682858141 +0000 UTC m=+1523.815986602" watchObservedRunningTime="2026-11-03 20:26:33.693462714 +0000 UTC m=+1523.826591175"
Nov 03 20:26:33.929518 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:33.929494    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:26:34.025973 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.024183    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-podinfo\") pod \"f9ec3c59-274d-4af5-a710-751cf5367d3c\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") "
Nov 03 20:26:34.025973 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.024301    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-workload-socket\") pod \"f9ec3c59-274d-4af5-a710-751cf5367d3c\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") "
Nov 03 20:26:34.025973 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.024360    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-token\" (UniqueName: \"kubernetes.io/projected/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-token\") pod \"f9ec3c59-274d-4af5-a710-751cf5367d3c\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") "
Nov 03 20:26:34.025973 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.024412    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-envoy\") pod \"f9ec3c59-274d-4af5-a710-751cf5367d3c\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") "
Nov 03 20:26:34.025973 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.024463    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9ec3c59-274d-4af5-a710-751cf5367d3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ec3c59-274d-4af5-a710-751cf5367d3c-istiod-ca-cert\") pod \"f9ec3c59-274d-4af5-a710-751cf5367d3c\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") "
Nov 03 20:26:34.025973 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.024533    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9ec3c59-274d-4af5-a710-751cf5367d3c-kube-api-access-w5dq5\" (UniqueName: \"kubernetes.io/projected/f9ec3c59-274d-4af5-a710-751cf5367d3c-kube-api-access-w5dq5\") pod \"f9ec3c59-274d-4af5-a710-751cf5367d3c\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") "
Nov 03 20:26:34.025973 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.024668    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-data\") pod \"f9ec3c59-274d-4af5-a710-751cf5367d3c\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") "
Nov 03 20:26:34.025973 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.024803    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-workload-certs\") pod \"f9ec3c59-274d-4af5-a710-751cf5367d3c\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") "
Nov 03 20:26:34.025973 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.024853    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-credential-socket\") pod \"f9ec3c59-274d-4af5-a710-751cf5367d3c\" (UID: \"f9ec3c59-274d-4af5-a710-751cf5367d3c\") "
Nov 03 20:26:34.031148 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.029462    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-data" pod "f9ec3c59-274d-4af5-a710-751cf5367d3c" (UID: "f9ec3c59-274d-4af5-a710-751cf5367d3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:34.031148 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.029954    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-workload-certs" pod "f9ec3c59-274d-4af5-a710-751cf5367d3c" (UID: "f9ec3c59-274d-4af5-a710-751cf5367d3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:34.031148 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.030069    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9ec3c59-274d-4af5-a710-751cf5367d3c-istiod-ca-cert" pod "f9ec3c59-274d-4af5-a710-751cf5367d3c" (UID: "f9ec3c59-274d-4af5-a710-751cf5367d3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:34.031424 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.031399    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-credential-socket" pod "f9ec3c59-274d-4af5-a710-751cf5367d3c" (UID: "f9ec3c59-274d-4af5-a710-751cf5367d3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:34.033822 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.033791    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ec3c59-274d-4af5-a710-751cf5367d3c-kube-api-access-w5dq5" pod "f9ec3c59-274d-4af5-a710-751cf5367d3c" (UID: "f9ec3c59-274d-4af5-a710-751cf5367d3c"). InnerVolumeSpecName "kube-api-access-w5dq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:34.040671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.040340    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-workload-socket" pod "f9ec3c59-274d-4af5-a710-751cf5367d3c" (UID: "f9ec3c59-274d-4af5-a710-751cf5367d3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:34.042961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.042919    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-podinfo" pod "f9ec3c59-274d-4af5-a710-751cf5367d3c" (UID: "f9ec3c59-274d-4af5-a710-751cf5367d3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:34.043068 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.042963    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-token" pod "f9ec3c59-274d-4af5-a710-751cf5367d3c" (UID: "f9ec3c59-274d-4af5-a710-751cf5367d3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:34.047262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.047235    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-envoy" pod "f9ec3c59-274d-4af5-a710-751cf5367d3c" (UID: "f9ec3c59-274d-4af5-a710-751cf5367d3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:34.099576 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.099546    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n"]
Nov 03 20:26:34.100250 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.100235    4421 cpu_manager.go:414] "containerMap: removing container" podUID="f9ec3c59-274d-4af5-a710-751cf5367d3c" containerName="istio-proxy"
Nov 03 20:26:34.100312 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.100251    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9ec3c59-274d-4af5-a710-751cf5367d3c" containerName="istio-proxy"
Nov 03 20:26:34.100312 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.100270    4421 cpu_manager.go:414] "containerMap: removing container" podUID="f9ec3c59-274d-4af5-a710-751cf5367d3c" containerName="istio-proxy"
Nov 03 20:26:34.100312 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.100275    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9ec3c59-274d-4af5-a710-751cf5367d3c" containerName="istio-proxy"
Nov 03 20:26:34.100421 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.100375    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9ec3c59-274d-4af5-a710-751cf5367d3c" containerName="istio-proxy"
Nov 03 20:26:34.100421 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.100386    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9ec3c59-274d-4af5-a710-751cf5367d3c" containerName="istio-proxy"
Nov 03 20:26:34.104395 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.104373    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n"
Nov 03 20:26:34.110014 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.109988    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-ba89e0\""
Nov 03 20:26:34.110158 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.110027    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-ba89e0-dockercfg-qw5kq\""
Nov 03 20:26:34.113368 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.113343    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n"]
Nov 03 20:26:34.127912 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.127876    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5dq5\" (UniqueName: \"kubernetes.io/projected/f9ec3c59-274d-4af5-a710-751cf5367d3c-kube-api-access-w5dq5\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:34.127912 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.127914    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:34.128122 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.127933    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:34.128122 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.127951    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:34.128122 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.127965    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:34.128122 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.127978    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:34.128122 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.127992    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:34.128122 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.128006    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9ec3c59-274d-4af5-a710-751cf5367d3c-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:34.128122 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.128019    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9ec3c59-274d-4af5-a710-751cf5367d3c-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:34.143844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.143814    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw"]
Nov 03 20:26:34.148274 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.148226    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw"
Nov 03 20:26:34.159105 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.159079    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw"]
Nov 03 20:26:34.231632 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.230156    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0672d75-59ba-490f-bc47-7b2f37ad0e1f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw\" (UID: \"e0672d75-59ba-490f-bc47-7b2f37ad0e1f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw"
Nov 03 20:26:34.231632 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.230283    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppbn4\" (UniqueName: \"kubernetes.io/projected/e0672d75-59ba-490f-bc47-7b2f37ad0e1f-kube-api-access-ppbn4\") pod \"payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw\" (UID: \"e0672d75-59ba-490f-bc47-7b2f37ad0e1f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw"
Nov 03 20:26:34.231632 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.230415    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ng8z\" (UniqueName: \"kubernetes.io/projected/f70b4f2d-2560-4801-8137-ad1d2d0adf4b-kube-api-access-8ng8z\") pod \"payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n\" (UID: \"f70b4f2d-2560-4801-8137-ad1d2d0adf4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n"
Nov 03 20:26:34.231632 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.230510    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f70b4f2d-2560-4801-8137-ad1d2d0adf4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n\" (UID: \"f70b4f2d-2560-4801-8137-ad1d2d0adf4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n"
Nov 03 20:26:34.398190 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.398062    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0672d75-59ba-490f-bc47-7b2f37ad0e1f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw\" (UID: \"e0672d75-59ba-490f-bc47-7b2f37ad0e1f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw"
Nov 03 20:26:34.398786 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.398660    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppbn4\" (UniqueName: \"kubernetes.io/projected/e0672d75-59ba-490f-bc47-7b2f37ad0e1f-kube-api-access-ppbn4\") pod \"payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw\" (UID: \"e0672d75-59ba-490f-bc47-7b2f37ad0e1f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw"
Nov 03 20:26:34.399556 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.399437    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ng8z\" (UniqueName: \"kubernetes.io/projected/f70b4f2d-2560-4801-8137-ad1d2d0adf4b-kube-api-access-8ng8z\") pod \"payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n\" (UID: \"f70b4f2d-2560-4801-8137-ad1d2d0adf4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n"
Nov 03 20:26:34.400002 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.399958    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f70b4f2d-2560-4801-8137-ad1d2d0adf4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n\" (UID: \"f70b4f2d-2560-4801-8137-ad1d2d0adf4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n"
Nov 03 20:26:34.406111 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.401919    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0672d75-59ba-490f-bc47-7b2f37ad0e1f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw\" (UID: \"e0672d75-59ba-490f-bc47-7b2f37ad0e1f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw"
Nov 03 20:26:34.408450 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.408426    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f70b4f2d-2560-4801-8137-ad1d2d0adf4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n\" (UID: \"f70b4f2d-2560-4801-8137-ad1d2d0adf4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n"
Nov 03 20:26:34.487975 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.487945    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppbn4\" (UniqueName: \"kubernetes.io/projected/e0672d75-59ba-490f-bc47-7b2f37ad0e1f-kube-api-access-ppbn4\") pod \"payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw\" (UID: \"e0672d75-59ba-490f-bc47-7b2f37ad0e1f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw"
Nov 03 20:26:34.489356 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.489333    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ng8z\" (UniqueName: \"kubernetes.io/projected/f70b4f2d-2560-4801-8137-ad1d2d0adf4b-kube-api-access-8ng8z\") pod \"payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n\" (UID: \"f70b4f2d-2560-4801-8137-ad1d2d0adf4b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n"
Nov 03 20:26:34.492562 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.492541    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw"
Nov 03 20:26:34.683381 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.682559    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"
Nov 03 20:26:34.685712 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.685549    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq" event={"ID":"f9ec3c59-274d-4af5-a710-751cf5367d3c","Type":"ContainerDied","Data":"fdc9bae017a96705e406195cc9c2fe377b15f6e23533a61ffa9c04a518a3dfd2"}
Nov 03 20:26:34.742024 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.738348    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n"
Nov 03 20:26:34.742024 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.738702    4421 scope.go:122] "RemoveContainer" containerID="25dabfd5874956d61f410ca631a998a8b81437b121d42e494c4e9b3d46e70abc"
Nov 03 20:26:34.862019 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.861953    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"]
Nov 03 20:26:34.866458 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.866426    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw"]
Nov 03 20:26:34.870691 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:34.870670    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2eee8c-openshift-default-7bb66d8f55-6tzlq"]
Nov 03 20:26:35.122848 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:26:35.122811    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70b4f2d_2560_4801_8137_ad1d2d0adf4b.slice/crio-72026110affaf6794700ce0ddc023a7d9492922094d51f92fb3855f489bc9b50 WatchSource:0}: Error finding container 72026110affaf6794700ce0ddc023a7d9492922094d51f92fb3855f489bc9b50: Status 404 returned error can't find the container with id 72026110affaf6794700ce0ddc023a7d9492922094d51f92fb3855f489bc9b50
Nov 03 20:26:35.123213 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:35.123188    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n"]
Nov 03 20:26:35.690490 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:35.690456    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n" event={"ID":"f70b4f2d-2560-4801-8137-ad1d2d0adf4b","Type":"ContainerStarted","Data":"b2ebf5ac7017bc5572cb9e5f318d62e5292ec93ec4b341e10ad1f8c1701dd18d"}
Nov 03 20:26:35.691018 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:35.690518    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n" event={"ID":"f70b4f2d-2560-4801-8137-ad1d2d0adf4b","Type":"ContainerStarted","Data":"72026110affaf6794700ce0ddc023a7d9492922094d51f92fb3855f489bc9b50"}
Nov 03 20:26:35.691018 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:35.690566    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n"
Nov 03 20:26:35.693274 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:35.693254    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw" event={"ID":"e0672d75-59ba-490f-bc47-7b2f37ad0e1f","Type":"ContainerStarted","Data":"0ef1bd0b156b4f6681d9698869e9ab4d526d8bd07290f51020d9c07252ba3b73"}
Nov 03 20:26:35.693376 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:35.693293    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw" event={"ID":"e0672d75-59ba-490f-bc47-7b2f37ad0e1f","Type":"ContainerStarted","Data":"c5418173bdf49e9c588f8ea411e0ee76edfae8fa349cf5d57ddf7262a2f2684c"}
Nov 03 20:26:35.693376 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:35.693333    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw"
Nov 03 20:26:35.711806 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:35.711767    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n" podStartSLOduration=1.711744034 podStartE2EDuration="1.711744034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:35.708534727 +0000 UTC m=+1525.841663187" watchObservedRunningTime="2026-11-03 20:26:35.711744034 +0000 UTC m=+1525.844872499"
Nov 03 20:26:35.730424 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:35.730315    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw" podStartSLOduration=1.730298071 podStartE2EDuration="1.730298071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:35.726928675 +0000 UTC m=+1525.860057138" watchObservedRunningTime="2026-11-03 20:26:35.730298071 +0000 UTC m=+1525.863426533"
Nov 03 20:26:36.018225 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:36.018148    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9ec3c59-274d-4af5-a710-751cf5367d3c" path="/var/lib/kubelet/pods/f9ec3c59-274d-4af5-a710-751cf5367d3c/volumes"
Nov 03 20:26:37.065084 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:37.065054    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl"]
Nov 03 20:26:37.065517 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:37.065243    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl" podUID="d3fd1d16-f04c-4807-8991-1987e5245fb8" containerName="payload-processing" containerID="cri-o://e6f765617667051d74514a60bdd41c7f974bb76af2acba589af39f827e77d94d" gracePeriod=30
Nov 03 20:26:37.093916 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:37.093882    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28"]
Nov 03 20:26:37.094262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:37.094229    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28" podUID="0d787a81-bb04-461a-861d-e81e1688d5a7" containerName="payload-pre-processing" containerID="cri-o://4f37a359adb5ae0177ad0fb2ed1f8cd2b58d73fe09ba4de538ee0337b828607f" gracePeriod=30
Nov 03 20:26:37.483235 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:37.464182    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q"]
Nov 03 20:26:37.483235 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:37.470172    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q"
Nov 03 20:26:37.483235 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:37.481618    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q"]
Nov 03 20:26:37.595661 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:37.489978    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-08a7bc-serving-cert\""
Nov 03 20:26:37.595661 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:37.517591    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz"
Nov 03 20:26:37.595661 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:37.519391    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f"
Nov 03 20:26:37.635686 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:37.631359    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q\" (UID: \"38f64ec2-df7a-4c4e-bdd5-d25f13125ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q"
Nov 03 20:26:37.635686 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:37.631488    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q\" (UID: \"38f64ec2-df7a-4c4e-bdd5-d25f13125ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q"
Nov 03 20:26:37.635686 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:37.631564    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q\" (UID: \"38f64ec2-df7a-4c4e-bdd5-d25f13125ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q"
Nov 03 20:26:37.635686 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:37.632476    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g46zn\" (UniqueName: \"kubernetes.io/projected/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-kube-api-access-g46zn\") pod \"maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q\" (UID: \"38f64ec2-df7a-4c4e-bdd5-d25f13125ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q"
Nov 03 20:26:37.747619 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:37.742909    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g46zn\" (UniqueName: \"kubernetes.io/projected/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-kube-api-access-g46zn\") pod \"maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q\" (UID: \"38f64ec2-df7a-4c4e-bdd5-d25f13125ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q"
Nov 03 20:26:37.747619 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:37.743189    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q\" (UID: \"38f64ec2-df7a-4c4e-bdd5-d25f13125ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q"
Nov 03 20:26:37.747619 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:37.743275    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q\" (UID: \"38f64ec2-df7a-4c4e-bdd5-d25f13125ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q"
Nov 03 20:26:37.747619 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:37.743321    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q\" (UID: \"38f64ec2-df7a-4c4e-bdd5-d25f13125ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q"
Nov 03 20:26:37.749466 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:26:37.749442    4421 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-serving-cert: secret "maas-api-e2e-shared-b-w1-08a7bc-serving-cert" not found
Nov 03 20:26:37.749741 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:26:37.749713    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-maas-api-tls podName:38f64ec2-df7a-4c4e-bdd5-d25f13125ad6 nodeName:}" failed. No retries permitted until 2026-11-03 20:26:38.249691947 +0000 UTC m=+1528.382820407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-maas-api-tls") pod "maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q" (UID: "38f64ec2-df7a-4c4e-bdd5-d25f13125ad6") : secret "maas-api-e2e-shared-b-w1-08a7bc-serving-cert" not found
Nov 03 20:26:37.751256 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:37.751213    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q\" (UID: \"38f64ec2-df7a-4c4e-bdd5-d25f13125ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q"
Nov 03 20:26:37.759998 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:37.759952    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q\" (UID: \"38f64ec2-df7a-4c4e-bdd5-d25f13125ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q"
Nov 03 20:26:37.785618 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:37.783491    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g46zn\" (UniqueName: \"kubernetes.io/projected/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-kube-api-access-g46zn\") pod \"maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q\" (UID: \"38f64ec2-df7a-4c4e-bdd5-d25f13125ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q"
Nov 03 20:26:37.789481 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:37.788650    4421 generic.go:356] "Generic (PLEG): container finished" podID="d3fd1d16-f04c-4807-8991-1987e5245fb8" containerID="e6f765617667051d74514a60bdd41c7f974bb76af2acba589af39f827e77d94d" exitCode=0
Nov 03 20:26:37.789481 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:37.788776    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl" event={"ID":"d3fd1d16-f04c-4807-8991-1987e5245fb8","Type":"ContainerDied","Data":"e6f765617667051d74514a60bdd41c7f974bb76af2acba589af39f827e77d94d"}
Nov 03 20:26:37.791194 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:37.791168    4421 generic.go:356] "Generic (PLEG): container finished" podID="0d787a81-bb04-461a-861d-e81e1688d5a7" containerID="4f37a359adb5ae0177ad0fb2ed1f8cd2b58d73fe09ba4de538ee0337b828607f" exitCode=0
Nov 03 20:26:37.791351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:37.791337    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28" event={"ID":"0d787a81-bb04-461a-861d-e81e1688d5a7","Type":"ContainerDied","Data":"4f37a359adb5ae0177ad0fb2ed1f8cd2b58d73fe09ba4de538ee0337b828607f"}
Nov 03 20:26:37.854518 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:37.854491    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl"
Nov 03 20:26:37.926500 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:37.926472    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28"
Nov 03 20:26:38.182685 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:38.179248    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d787a81-bb04-461a-861d-e81e1688d5a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d787a81-bb04-461a-861d-e81e1688d5a7-plugins-config-volume\") pod \"0d787a81-bb04-461a-861d-e81e1688d5a7\" (UID: \"0d787a81-bb04-461a-861d-e81e1688d5a7\") "
Nov 03 20:26:38.182685 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:38.179332    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3fd1d16-f04c-4807-8991-1987e5245fb8-kube-api-access-57wsq\" (UniqueName: \"kubernetes.io/projected/d3fd1d16-f04c-4807-8991-1987e5245fb8-kube-api-access-57wsq\") pod \"d3fd1d16-f04c-4807-8991-1987e5245fb8\" (UID: \"d3fd1d16-f04c-4807-8991-1987e5245fb8\") "
Nov 03 20:26:38.182685 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:38.179377    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d787a81-bb04-461a-861d-e81e1688d5a7-kube-api-access-r99h5\" (UniqueName: \"kubernetes.io/projected/0d787a81-bb04-461a-861d-e81e1688d5a7-kube-api-access-r99h5\") pod \"0d787a81-bb04-461a-861d-e81e1688d5a7\" (UID: \"0d787a81-bb04-461a-861d-e81e1688d5a7\") "
Nov 03 20:26:38.182685 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:38.179476    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3fd1d16-f04c-4807-8991-1987e5245fb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3fd1d16-f04c-4807-8991-1987e5245fb8-plugins-config-volume\") pod \"d3fd1d16-f04c-4807-8991-1987e5245fb8\" (UID: \"d3fd1d16-f04c-4807-8991-1987e5245fb8\") "
Nov 03 20:26:38.188625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:38.183625    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3fd1d16-f04c-4807-8991-1987e5245fb8-plugins-config-volume" pod "d3fd1d16-f04c-4807-8991-1987e5245fb8" (UID: "d3fd1d16-f04c-4807-8991-1987e5245fb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:38.195143 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:38.194808    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d787a81-bb04-461a-861d-e81e1688d5a7-plugins-config-volume" pod "0d787a81-bb04-461a-861d-e81e1688d5a7" (UID: "0d787a81-bb04-461a-861d-e81e1688d5a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:38.212463 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:38.212103    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3fd1d16-f04c-4807-8991-1987e5245fb8-kube-api-access-57wsq" pod "d3fd1d16-f04c-4807-8991-1987e5245fb8" (UID: "d3fd1d16-f04c-4807-8991-1987e5245fb8"). InnerVolumeSpecName "kube-api-access-57wsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:38.282997 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:38.282955    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q\" (UID: \"38f64ec2-df7a-4c4e-bdd5-d25f13125ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q"
Nov 03 20:26:38.283778 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:38.283751    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d787a81-bb04-461a-861d-e81e1688d5a7-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:38.284003 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:38.283988    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57wsq\" (UniqueName: \"kubernetes.io/projected/d3fd1d16-f04c-4807-8991-1987e5245fb8-kube-api-access-57wsq\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:38.284148 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:38.284135    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3fd1d16-f04c-4807-8991-1987e5245fb8-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:38.323241 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:38.323194    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q\" (UID: \"38f64ec2-df7a-4c4e-bdd5-d25f13125ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q"
Nov 03 20:26:38.347641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:38.346994    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d787a81-bb04-461a-861d-e81e1688d5a7-kube-api-access-r99h5" pod "0d787a81-bb04-461a-861d-e81e1688d5a7" (UID: "0d787a81-bb04-461a-861d-e81e1688d5a7"). InnerVolumeSpecName "kube-api-access-r99h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:38.388640 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:38.385949    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r99h5\" (UniqueName: \"kubernetes.io/projected/0d787a81-bb04-461a-861d-e81e1688d5a7-kube-api-access-r99h5\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:38.416678 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:38.414899    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q"
Nov 03 20:26:38.717361 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:38.717324    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q"]
Nov 03 20:26:38.717581 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:26:38.717560    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f64ec2_df7a_4c4e_bdd5_d25f13125ad6.slice/crio-2c973ce6da576727ea988c0f4238a236de3a3ab55f66ae27b091119a48c60f9b WatchSource:0}: Error finding container 2c973ce6da576727ea988c0f4238a236de3a3ab55f66ae27b091119a48c60f9b: Status 404 returned error can't find the container with id 2c973ce6da576727ea988c0f4238a236de3a3ab55f66ae27b091119a48c60f9b
Nov 03 20:26:38.800522 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:38.800485    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl" event={"ID":"d3fd1d16-f04c-4807-8991-1987e5245fb8","Type":"ContainerDied","Data":"13c650ba7931dbbda094c77d65b3cb475d6a4e5ed502a1d49a4b1472c3d7cf42"}
Nov 03 20:26:38.800759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:38.800525    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl"
Nov 03 20:26:38.800759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:38.800560    4421 scope.go:122] "RemoveContainer" containerID="e6f765617667051d74514a60bdd41c7f974bb76af2acba589af39f827e77d94d"
Nov 03 20:26:38.802097 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:38.802067    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28"
Nov 03 20:26:38.802231 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:38.802119    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28" event={"ID":"0d787a81-bb04-461a-861d-e81e1688d5a7","Type":"ContainerDied","Data":"a4f12321a3ffc3025508a996ae0b0a27652de5f7ddf8ec087eda6dbed33492a6"}
Nov 03 20:26:38.803842 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:38.803811    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q" event={"ID":"38f64ec2-df7a-4c4e-bdd5-d25f13125ad6","Type":"ContainerStarted","Data":"2c973ce6da576727ea988c0f4238a236de3a3ab55f66ae27b091119a48c60f9b"}
Nov 03 20:26:38.813864 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:38.813847    4421 scope.go:122] "RemoveContainer" containerID="4f37a359adb5ae0177ad0fb2ed1f8cd2b58d73fe09ba4de538ee0337b828607f"
Nov 03 20:26:38.825311 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:38.825286    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl"]
Nov 03 20:26:38.830669 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:38.830589    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1142fd-66fd5b6bd4-2cffl"]
Nov 03 20:26:38.851267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:38.843017    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28"]
Nov 03 20:26:38.890623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:38.883285    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1142fd-76f76b5df9-54r28"]
Nov 03 20:26:39.230174 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:39.230146    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh"]
Nov 03 20:26:39.230927 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:39.230908    4421 cpu_manager.go:414] "containerMap: removing container" podUID="d3fd1d16-f04c-4807-8991-1987e5245fb8" containerName="payload-processing"
Nov 03 20:26:39.230993 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:39.230930    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3fd1d16-f04c-4807-8991-1987e5245fb8" containerName="payload-processing"
Nov 03 20:26:39.230993 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:39.230968    4421 cpu_manager.go:414] "containerMap: removing container" podUID="0d787a81-bb04-461a-861d-e81e1688d5a7" containerName="payload-pre-processing"
Nov 03 20:26:39.230993 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:39.230973    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d787a81-bb04-461a-861d-e81e1688d5a7" containerName="payload-pre-processing"
Nov 03 20:26:39.231092 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:39.231077    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3fd1d16-f04c-4807-8991-1987e5245fb8" containerName="payload-processing"
Nov 03 20:26:39.231128 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:39.231091    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d787a81-bb04-461a-861d-e81e1688d5a7" containerName="payload-pre-processing"
Nov 03 20:26:39.235924 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:39.235904    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh"
Nov 03 20:26:39.240852 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:39.240817    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-bab2c3\""
Nov 03 20:26:39.241560 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:39.240968    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-bab2c3-dockercfg-tjg8h\""
Nov 03 20:26:39.247638 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:39.247582    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh"]
Nov 03 20:26:39.475677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:39.430674    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb20c2e-30b6-4bdd-9b91-fc996180e96d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh\" (UID: \"6eb20c2e-30b6-4bdd-9b91-fc996180e96d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh"
Nov 03 20:26:39.475677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:39.430860    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs82l\" (UniqueName: \"kubernetes.io/projected/6eb20c2e-30b6-4bdd-9b91-fc996180e96d-kube-api-access-vs82l\") pod \"payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh\" (UID: \"6eb20c2e-30b6-4bdd-9b91-fc996180e96d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh"
Nov 03 20:26:39.548449 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:39.533742    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb20c2e-30b6-4bdd-9b91-fc996180e96d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh\" (UID: \"6eb20c2e-30b6-4bdd-9b91-fc996180e96d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh"
Nov 03 20:26:39.548449 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:39.533906    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs82l\" (UniqueName: \"kubernetes.io/projected/6eb20c2e-30b6-4bdd-9b91-fc996180e96d-kube-api-access-vs82l\") pod \"payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh\" (UID: \"6eb20c2e-30b6-4bdd-9b91-fc996180e96d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh"
Nov 03 20:26:39.548449 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:39.535152    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb20c2e-30b6-4bdd-9b91-fc996180e96d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh\" (UID: \"6eb20c2e-30b6-4bdd-9b91-fc996180e96d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh"
Nov 03 20:26:39.606791 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:39.593954    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs82l\" (UniqueName: \"kubernetes.io/projected/6eb20c2e-30b6-4bdd-9b91-fc996180e96d-kube-api-access-vs82l\") pod \"payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh\" (UID: \"6eb20c2e-30b6-4bdd-9b91-fc996180e96d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh"
Nov 03 20:26:39.814275 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:39.814179    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q" event={"ID":"38f64ec2-df7a-4c4e-bdd5-d25f13125ad6","Type":"ContainerStarted","Data":"29fcbe7958e6c047827c58676aa8a4b9d6747d50fc67531a0b6cddc61e52979b"}
Nov 03 20:26:39.814275 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:39.814253    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q"
Nov 03 20:26:39.865631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:39.853290    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q" podStartSLOduration=2.394607938 podStartE2EDuration="2.853269403s" totalImagesPullingTime="458.661465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:39.852186985 +0000 UTC m=+1529.985315449" watchObservedRunningTime="2026-11-03 20:26:39.853269403 +0000 UTC m=+1529.986397870"
Nov 03 20:26:39.890328 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:39.868900    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh"
Nov 03 20:26:40.099335 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:40.098322    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d787a81-bb04-461a-861d-e81e1688d5a7" path="/var/lib/kubelet/pods/0d787a81-bb04-461a-861d-e81e1688d5a7/volumes"
Nov 03 20:26:40.099335 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:40.098803    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3fd1d16-f04c-4807-8991-1987e5245fb8" path="/var/lib/kubelet/pods/d3fd1d16-f04c-4807-8991-1987e5245fb8/volumes"
Nov 03 20:26:40.234953 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:40.234925    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh"]
Nov 03 20:26:40.235818 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:26:40.235792    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb20c2e_30b6_4bdd_9b91_fc996180e96d.slice/crio-2d0438dc4b22a6067e20bdc7dc4cc707bdfb68ac85e0e42d0928453688d27506 WatchSource:0}: Error finding container 2d0438dc4b22a6067e20bdc7dc4cc707bdfb68ac85e0e42d0928453688d27506: Status 404 returned error can't find the container with id 2d0438dc4b22a6067e20bdc7dc4cc707bdfb68ac85e0e42d0928453688d27506
Nov 03 20:26:40.699737 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:40.699709    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p"
Nov 03 20:26:40.825885 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:40.825853    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh" event={"ID":"6eb20c2e-30b6-4bdd-9b91-fc996180e96d","Type":"ContainerStarted","Data":"a4f46e6818b2d478abea089fd894a83d37fafdddee26a1eddd5941ab4fb2e5d3"}
Nov 03 20:26:40.826040 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:40.825910    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh" event={"ID":"6eb20c2e-30b6-4bdd-9b91-fc996180e96d","Type":"ContainerStarted","Data":"2d0438dc4b22a6067e20bdc7dc4cc707bdfb68ac85e0e42d0928453688d27506"}
Nov 03 20:26:40.826237 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:40.826221    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh"
Nov 03 20:26:40.848681 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:40.848632    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh" podStartSLOduration=1.848612385 podStartE2EDuration="1.848612385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:40.843724502 +0000 UTC m=+1530.976852963" watchObservedRunningTime="2026-11-03 20:26:40.848612385 +0000 UTC m=+1530.981740843"
Nov 03 20:26:41.220667 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:41.218585    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg"]
Nov 03 20:26:41.224863 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:41.224842    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg"
Nov 03 20:26:41.236474 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:41.236370    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-ba89e0-serving-cert\""
Nov 03 20:26:41.251477 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:41.251445    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg"]
Nov 03 20:26:41.407277 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:41.407240    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg\" (UID: \"bcc56ff3-69b2-4de1-a481-c5dc245ebe83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg"
Nov 03 20:26:41.408341 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:41.408316    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg\" (UID: \"bcc56ff3-69b2-4de1-a481-c5dc245ebe83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg"
Nov 03 20:26:41.408539 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:41.408519    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt2db\" (UniqueName: \"kubernetes.io/projected/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-kube-api-access-lt2db\") pod \"maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg\" (UID: \"bcc56ff3-69b2-4de1-a481-c5dc245ebe83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg"
Nov 03 20:26:41.421622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:41.408855    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg\" (UID: \"bcc56ff3-69b2-4de1-a481-c5dc245ebe83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg"
Nov 03 20:26:41.535236 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:41.530294    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg\" (UID: \"bcc56ff3-69b2-4de1-a481-c5dc245ebe83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg"
Nov 03 20:26:41.535236 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:41.530344    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt2db\" (UniqueName: \"kubernetes.io/projected/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-kube-api-access-lt2db\") pod \"maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg\" (UID: \"bcc56ff3-69b2-4de1-a481-c5dc245ebe83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg"
Nov 03 20:26:41.535236 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:41.530543    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg\" (UID: \"bcc56ff3-69b2-4de1-a481-c5dc245ebe83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg"
Nov 03 20:26:41.535236 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:41.530739    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg\" (UID: \"bcc56ff3-69b2-4de1-a481-c5dc245ebe83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg"
Nov 03 20:26:41.547615 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:41.541376    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg\" (UID: \"bcc56ff3-69b2-4de1-a481-c5dc245ebe83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg"
Nov 03 20:26:41.547615 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:41.544789    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg\" (UID: \"bcc56ff3-69b2-4de1-a481-c5dc245ebe83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg"
Nov 03 20:26:41.598875 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:41.598172    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg\" (UID: \"bcc56ff3-69b2-4de1-a481-c5dc245ebe83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg"
Nov 03 20:26:41.600034 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:41.599984    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt2db\" (UniqueName: \"kubernetes.io/projected/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-kube-api-access-lt2db\") pod \"maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg\" (UID: \"bcc56ff3-69b2-4de1-a481-c5dc245ebe83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg"
Nov 03 20:26:41.862690 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:41.859372    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg"
Nov 03 20:26:42.293103 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:42.293074    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg"]
Nov 03 20:26:42.293502 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:26:42.293476    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc56ff3_69b2_4de1_a481_c5dc245ebe83.slice/crio-54b92a47641673fe8b53445e7cfca44aa6fac09882c495cad2832d18bde00b3d WatchSource:0}: Error finding container 54b92a47641673fe8b53445e7cfca44aa6fac09882c495cad2832d18bde00b3d: Status 404 returned error can't find the container with id 54b92a47641673fe8b53445e7cfca44aa6fac09882c495cad2832d18bde00b3d
Nov 03 20:26:42.646840 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:42.646288    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj"
Nov 03 20:26:42.646840 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:42.646634    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz"
Nov 03 20:26:43.697108 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:43.688036    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg" event={"ID":"bcc56ff3-69b2-4de1-a481-c5dc245ebe83","Type":"ContainerStarted","Data":"54b92a47641673fe8b53445e7cfca44aa6fac09882c495cad2832d18bde00b3d"}
Nov 03 20:26:44.267768 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:44.267727    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 20:26:44.418819 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:44.408106    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:44.441660 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:44.436716    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:44.441660 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:44.440717    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:44.582680 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:44.580697    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:44.607624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:44.605731    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 03 20:26:45.102152 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:45.102119    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg" event={"ID":"bcc56ff3-69b2-4de1-a481-c5dc245ebe83","Type":"ContainerStarted","Data":"83a2baabbe9d3cf7b7cb88d704402ff4c58e6ff628599f7d7031026455bf679f"}
Nov 03 20:26:45.103790 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:45.103765    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg"
Nov 03 20:26:45.157463 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:45.157403    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg" podStartSLOduration=2.910867811 podStartE2EDuration="4.157380448s" totalImagesPullingTime="1.246512637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:45.131132376 +0000 UTC m=+1535.264260947" watchObservedRunningTime="2026-11-03 20:26:45.157380448 +0000 UTC m=+1535.290508912"
Nov 03 20:26:45.284623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:45.278735    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:45.284623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:45.280518    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:26:45.454577 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:45.446115    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:45.454577 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:45.447515    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:26:45.454577 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:45.448357    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:26:45.454577 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:45.448447    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:26:45.455627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:45.455036    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:45.459624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:45.456178    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:26:45.588078 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:45.588034    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:45.768677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:45.695052    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:26:45.995844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:45.991195    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q"
Nov 03 20:26:46.010631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:46.002731    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 03 20:26:46.045844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:46.045740    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:26:46.240650 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:46.226767    4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:46.524624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:46.522015    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:46.540031 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:46.534938    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:46.582628 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:46.571571    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 20:26:46.856695 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:46.666586    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:46.856695 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:46.666724    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:26:47.080623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:47.076488    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 03 20:26:47.588622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:47.553722    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:26:47.588622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:47.554134    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:26:47.709783 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:26:47.709739    4421 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.107s"
Nov 03 20:26:47.709957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:47.709826    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n"
Nov 03 20:26:47.709957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:47.709886    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw"
Nov 03 20:26:47.709957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:47.709937    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:26:47.726973 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:47.726945    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:26:47.757651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:47.753906    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:26:47.797749 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:47.797723    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:26:48.002063 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:48.002028    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:26:48.437044 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:48.436006    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:26:49.632410 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:26:49.632366    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:26:51.936775 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:51.936748    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh"
Nov 03 20:26:53.119046 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:53.119013    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq"]
Nov 03 20:26:53.124747 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:53.124727    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq"
Nov 03 20:26:53.134694 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:53.134650    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq"]
Nov 03 20:26:53.136630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:53.136588    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-3211a0\""
Nov 03 20:26:53.293824 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:53.293782    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f497057c-a42e-452f-87f1-f4d197459616-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq\" (UID: \"f497057c-a42e-452f-87f1-f4d197459616\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq"
Nov 03 20:26:53.344748 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:53.344715    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5742\" (UniqueName: \"kubernetes.io/projected/f497057c-a42e-452f-87f1-f4d197459616-kube-api-access-x5742\") pod \"payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq\" (UID: \"f497057c-a42e-452f-87f1-f4d197459616\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq"
Nov 03 20:26:53.446385 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:53.446349    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f497057c-a42e-452f-87f1-f4d197459616-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq\" (UID: \"f497057c-a42e-452f-87f1-f4d197459616\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq"
Nov 03 20:26:53.446876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:53.446849    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5742\" (UniqueName: \"kubernetes.io/projected/f497057c-a42e-452f-87f1-f4d197459616-kube-api-access-x5742\") pod \"payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq\" (UID: \"f497057c-a42e-452f-87f1-f4d197459616\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq"
Nov 03 20:26:53.447438 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:53.447411    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f497057c-a42e-452f-87f1-f4d197459616-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq\" (UID: \"f497057c-a42e-452f-87f1-f4d197459616\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq"
Nov 03 20:26:53.498745 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:53.498709    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5742\" (UniqueName: \"kubernetes.io/projected/f497057c-a42e-452f-87f1-f4d197459616-kube-api-access-x5742\") pod \"payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq\" (UID: \"f497057c-a42e-452f-87f1-f4d197459616\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq"
Nov 03 20:26:53.544419 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:53.544391    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg"
Nov 03 20:26:53.834707 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:53.833390    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq"
Nov 03 20:26:54.227885 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:54.227850    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq"]
Nov 03 20:26:54.231839 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:26:54.231807    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf497057c_a42e_452f_87f1_f4d197459616.slice/crio-bcf03279203542401acc1e4edaff6d6a772f93ad78687fd07d30d8f51d1c01b7 WatchSource:0}: Error finding container bcf03279203542401acc1e4edaff6d6a772f93ad78687fd07d30d8f51d1c01b7: Status 404 returned error can't find the container with id bcf03279203542401acc1e4edaff6d6a772f93ad78687fd07d30d8f51d1c01b7
Nov 03 20:26:54.895233 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:54.895199    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq" event={"ID":"f497057c-a42e-452f-87f1-f4d197459616","Type":"ContainerStarted","Data":"cfa22993e7b50bcf76fe4fb0e3e0a25605d81f3b64aa88a1b5d9d103c442aa58"}
Nov 03 20:26:54.895406 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:54.895258    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq" event={"ID":"f497057c-a42e-452f-87f1-f4d197459616","Type":"ContainerStarted","Data":"bcf03279203542401acc1e4edaff6d6a772f93ad78687fd07d30d8f51d1c01b7"}
Nov 03 20:26:54.895406 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:54.895285    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq"
Nov 03 20:26:54.916272 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:54.916218    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq" podStartSLOduration=1.916204065 podStartE2EDuration="1.916204065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:54.912847817 +0000 UTC m=+1545.045976279" watchObservedRunningTime="2026-11-03 20:26:54.916204065 +0000 UTC m=+1545.049332527"
Nov 03 20:26:55.564111 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:55.564081    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg"]
Nov 03 20:26:55.568795 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:55.568776    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg"
Nov 03 20:26:55.578827 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:55.578668    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-30e9de\""
Nov 03 20:26:55.582054 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:55.579936    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-30e9de-dockercfg-m6gp5\""
Nov 03 20:26:55.582462 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:55.582437    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg"]
Nov 03 20:26:55.680904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:55.680865    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwhkw\" (UniqueName: \"kubernetes.io/projected/bd5b0764-48b8-4909-b75d-8f4b8575b015-kube-api-access-kwhkw\") pod \"payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg\" (UID: \"bd5b0764-48b8-4909-b75d-8f4b8575b015\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg"
Nov 03 20:26:55.681378 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:55.681352    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd5b0764-48b8-4909-b75d-8f4b8575b015-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg\" (UID: \"bd5b0764-48b8-4909-b75d-8f4b8575b015\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg"
Nov 03 20:26:55.793633 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:55.790378    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwhkw\" (UniqueName: \"kubernetes.io/projected/bd5b0764-48b8-4909-b75d-8f4b8575b015-kube-api-access-kwhkw\") pod \"payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg\" (UID: \"bd5b0764-48b8-4909-b75d-8f4b8575b015\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg"
Nov 03 20:26:55.793633 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:55.790562    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd5b0764-48b8-4909-b75d-8f4b8575b015-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg\" (UID: \"bd5b0764-48b8-4909-b75d-8f4b8575b015\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg"
Nov 03 20:26:55.793633 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:55.791665    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd5b0764-48b8-4909-b75d-8f4b8575b015-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg\" (UID: \"bd5b0764-48b8-4909-b75d-8f4b8575b015\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg"
Nov 03 20:26:55.858665 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:55.856618    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwhkw\" (UniqueName: \"kubernetes.io/projected/bd5b0764-48b8-4909-b75d-8f4b8575b015-kube-api-access-kwhkw\") pod \"payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg\" (UID: \"bd5b0764-48b8-4909-b75d-8f4b8575b015\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg"
Nov 03 20:26:55.904028 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:55.903992    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg"
Nov 03 20:26:56.327046 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:26:56.327010    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd5b0764_48b8_4909_b75d_8f4b8575b015.slice/crio-3f21ea9bc7fcb2927cb8823044849f8c671de0ff506be55522dbcd8f64a6fd75 WatchSource:0}: Error finding container 3f21ea9bc7fcb2927cb8823044849f8c671de0ff506be55522dbcd8f64a6fd75: Status 404 returned error can't find the container with id 3f21ea9bc7fcb2927cb8823044849f8c671de0ff506be55522dbcd8f64a6fd75
Nov 03 20:26:56.327773 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:56.327710    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg"]
Nov 03 20:26:56.990629 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:56.990553    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg" event={"ID":"bd5b0764-48b8-4909-b75d-8f4b8575b015","Type":"ContainerStarted","Data":"b97049446b49a88bea959b1ab8e2673fe52d575369b5acefa6c1738718930d5b"}
Nov 03 20:26:56.991044 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:56.990642    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg" event={"ID":"bd5b0764-48b8-4909-b75d-8f4b8575b015","Type":"ContainerStarted","Data":"3f21ea9bc7fcb2927cb8823044849f8c671de0ff506be55522dbcd8f64a6fd75"}
Nov 03 20:26:56.991044 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:56.990674    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg"
Nov 03 20:26:57.073498 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:26:57.072764    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg" podStartSLOduration=2.072743413 podStartE2EDuration="2.072743413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:57.015198381 +0000 UTC m=+1547.148326845" watchObservedRunningTime="2026-11-03 20:26:57.072743413 +0000 UTC m=+1547.205871875"
Nov 03 20:27:01.802749 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:01.802714    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2"]
Nov 03 20:27:01.808110 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:01.808087    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2"
Nov 03 20:27:01.816453 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:01.816426    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-3211a0-serving-cert\""
Nov 03 20:27:01.825355 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:01.824675    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2"]
Nov 03 20:27:01.917062 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:01.917031    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-metrics-certs\") pod \"maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2\" (UID: \"bac7a1a2-4692-4b09-9bb8-1d3faaa02692\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2"
Nov 03 20:27:01.917062 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:01.917064    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cbs2\" (UniqueName: \"kubernetes.io/projected/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-kube-api-access-5cbs2\") pod \"maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2\" (UID: \"bac7a1a2-4692-4b09-9bb8-1d3faaa02692\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2"
Nov 03 20:27:01.917445 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:01.917421    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2\" (UID: \"bac7a1a2-4692-4b09-9bb8-1d3faaa02692\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2"
Nov 03 20:27:01.917522 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:01.917498    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2\" (UID: \"bac7a1a2-4692-4b09-9bb8-1d3faaa02692\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2"
Nov 03 20:27:02.023783 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:02.021840    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2\" (UID: \"bac7a1a2-4692-4b09-9bb8-1d3faaa02692\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2"
Nov 03 20:27:02.023783 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:02.021932    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2\" (UID: \"bac7a1a2-4692-4b09-9bb8-1d3faaa02692\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2"
Nov 03 20:27:02.023783 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:02.021989    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-metrics-certs\") pod \"maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2\" (UID: \"bac7a1a2-4692-4b09-9bb8-1d3faaa02692\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2"
Nov 03 20:27:02.023783 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:02.022019    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cbs2\" (UniqueName: \"kubernetes.io/projected/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-kube-api-access-5cbs2\") pod \"maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2\" (UID: \"bac7a1a2-4692-4b09-9bb8-1d3faaa02692\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2"
Nov 03 20:27:02.023783 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:02.023327    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2\" (UID: \"bac7a1a2-4692-4b09-9bb8-1d3faaa02692\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2"
Nov 03 20:27:02.055717 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:02.049051    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2\" (UID: \"bac7a1a2-4692-4b09-9bb8-1d3faaa02692\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2"
Nov 03 20:27:02.080623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:02.077468    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cbs2\" (UniqueName: \"kubernetes.io/projected/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-kube-api-access-5cbs2\") pod \"maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2\" (UID: \"bac7a1a2-4692-4b09-9bb8-1d3faaa02692\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2"
Nov 03 20:27:02.103679 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:02.088303    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-metrics-certs\") pod \"maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2\" (UID: \"bac7a1a2-4692-4b09-9bb8-1d3faaa02692\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2"
Nov 03 20:27:02.152045 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:02.151977    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2"
Nov 03 20:27:02.525390 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:02.525358    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2"]
Nov 03 20:27:02.526699 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:27:02.526652    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac7a1a2_4692_4b09_9bb8_1d3faaa02692.slice/crio-9831ca3c71c1af8e55f9b4773cf052c613a7fab88cb7c591761a6860604152c1 WatchSource:0}: Error finding container 9831ca3c71c1af8e55f9b4773cf052c613a7fab88cb7c591761a6860604152c1: Status 404 returned error can't find the container with id 9831ca3c71c1af8e55f9b4773cf052c613a7fab88cb7c591761a6860604152c1
Nov 03 20:27:03.152235 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:03.152207    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2" event={"ID":"bac7a1a2-4692-4b09-9bb8-1d3faaa02692","Type":"ContainerStarted","Data":"9831ca3c71c1af8e55f9b4773cf052c613a7fab88cb7c591761a6860604152c1"}
Nov 03 20:27:04.164779 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:04.164747    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2" event={"ID":"bac7a1a2-4692-4b09-9bb8-1d3faaa02692","Type":"ContainerStarted","Data":"b626f0f0a542bb50969d3a88f5ab990fbb79fad6f7f36cb75f5e282f7fa0c728"}
Nov 03 20:27:04.165280 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:04.164828    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2"
Nov 03 20:27:04.259620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:04.248055    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2" podStartSLOduration=2.700682819 podStartE2EDuration="3.248033162s" totalImagesPullingTime="547.350343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:04.203932203 +0000 UTC m=+1554.337060666" watchObservedRunningTime="2026-11-03 20:27:04.248033162 +0000 UTC m=+1554.381161624"
Nov 03 20:27:05.985753 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:05.985723    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq"
Nov 03 20:27:08.053493 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:08.053462    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg"
Nov 03 20:27:08.742989 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:08.742953    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7"]
Nov 03 20:27:08.749124 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:08.749102    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7"
Nov 03 20:27:08.754273 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:08.754246    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-99a8fe\""
Nov 03 20:27:08.758787 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:08.758766    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7"]
Nov 03 20:27:08.777808 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:08.777782    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p"]
Nov 03 20:27:08.782258 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:08.782238    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p"
Nov 03 20:27:08.792290 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:08.790917    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-99a8fe-dockercfg-8rqnv\""
Nov 03 20:27:08.811735 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:08.811706    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p"]
Nov 03 20:27:08.949416 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:08.949382    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9smpw\" (UniqueName: \"kubernetes.io/projected/a872c432-f38a-437a-904e-26bd73eaea99-kube-api-access-9smpw\") pod \"payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7\" (UID: \"a872c432-f38a-437a-904e-26bd73eaea99\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7"
Nov 03 20:27:08.949627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:08.949476    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp66t\" (UniqueName: \"kubernetes.io/projected/0f074459-f167-446e-9acf-075cdbfc4ef0-kube-api-access-lp66t\") pod \"payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p\" (UID: \"0f074459-f167-446e-9acf-075cdbfc4ef0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p"
Nov 03 20:27:08.949627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:08.949583    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a872c432-f38a-437a-904e-26bd73eaea99-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7\" (UID: \"a872c432-f38a-437a-904e-26bd73eaea99\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7"
Nov 03 20:27:08.949755 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:08.949637    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f074459-f167-446e-9acf-075cdbfc4ef0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p\" (UID: \"0f074459-f167-446e-9acf-075cdbfc4ef0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p"
Nov 03 20:27:09.060691 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:09.060577    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a872c432-f38a-437a-904e-26bd73eaea99-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7\" (UID: \"a872c432-f38a-437a-904e-26bd73eaea99\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7"
Nov 03 20:27:09.089767 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:09.089729    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f074459-f167-446e-9acf-075cdbfc4ef0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p\" (UID: \"0f074459-f167-446e-9acf-075cdbfc4ef0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p"
Nov 03 20:27:09.090217 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:09.090196    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9smpw\" (UniqueName: \"kubernetes.io/projected/a872c432-f38a-437a-904e-26bd73eaea99-kube-api-access-9smpw\") pod \"payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7\" (UID: \"a872c432-f38a-437a-904e-26bd73eaea99\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7"
Nov 03 20:27:09.090469 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:09.090453    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp66t\" (UniqueName: \"kubernetes.io/projected/0f074459-f167-446e-9acf-075cdbfc4ef0-kube-api-access-lp66t\") pod \"payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p\" (UID: \"0f074459-f167-446e-9acf-075cdbfc4ef0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p"
Nov 03 20:27:09.091187 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:09.072202    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a872c432-f38a-437a-904e-26bd73eaea99-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7\" (UID: \"a872c432-f38a-437a-904e-26bd73eaea99\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7"
Nov 03 20:27:09.092243 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:09.092218    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f074459-f167-446e-9acf-075cdbfc4ef0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p\" (UID: \"0f074459-f167-446e-9acf-075cdbfc4ef0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p"
Nov 03 20:27:09.170633 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:09.170418    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9smpw\" (UniqueName: \"kubernetes.io/projected/a872c432-f38a-437a-904e-26bd73eaea99-kube-api-access-9smpw\") pod \"payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7\" (UID: \"a872c432-f38a-437a-904e-26bd73eaea99\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7"
Nov 03 20:27:09.195623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:09.171211    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp66t\" (UniqueName: \"kubernetes.io/projected/0f074459-f167-446e-9acf-075cdbfc4ef0-kube-api-access-lp66t\") pod \"payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p\" (UID: \"0f074459-f167-446e-9acf-075cdbfc4ef0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p"
Nov 03 20:27:09.367279 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:09.367188    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7"
Nov 03 20:27:09.438786 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:09.438541    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p"
Nov 03 20:27:09.661519 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:27:09.661399    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda872c432_f38a_437a_904e_26bd73eaea99.slice/crio-194a1e8676c7a1e4f8fc87b61eb25a71a8700f857fbd9ab3be89fd70cb090d33 WatchSource:0}: Error finding container 194a1e8676c7a1e4f8fc87b61eb25a71a8700f857fbd9ab3be89fd70cb090d33: Status 404 returned error can't find the container with id 194a1e8676c7a1e4f8fc87b61eb25a71a8700f857fbd9ab3be89fd70cb090d33
Nov 03 20:27:09.663329 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:09.663300    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7"]
Nov 03 20:27:09.823893 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:27:09.823866    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f074459_f167_446e_9acf_075cdbfc4ef0.slice/crio-5e66415c1e207c935a9a84cafb57c8ccd10cf723213a5a13f69ff53a4461a081 WatchSource:0}: Error finding container 5e66415c1e207c935a9a84cafb57c8ccd10cf723213a5a13f69ff53a4461a081: Status 404 returned error can't find the container with id 5e66415c1e207c935a9a84cafb57c8ccd10cf723213a5a13f69ff53a4461a081
Nov 03 20:27:09.824173 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:09.824153    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p"]
Nov 03 20:27:10.181479 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:10.181450    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2"
Nov 03 20:27:10.275624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:10.274786    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p" event={"ID":"0f074459-f167-446e-9acf-075cdbfc4ef0","Type":"ContainerStarted","Data":"36dc09b47a33fab5e1e4d9eed511843c3bb9252d16c2bc25cd03892479f4ea62"}
Nov 03 20:27:10.275624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:10.274855    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p" event={"ID":"0f074459-f167-446e-9acf-075cdbfc4ef0","Type":"ContainerStarted","Data":"5e66415c1e207c935a9a84cafb57c8ccd10cf723213a5a13f69ff53a4461a081"}
Nov 03 20:27:10.275624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:10.275089    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p"
Nov 03 20:27:10.278640 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:10.276989    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7" event={"ID":"a872c432-f38a-437a-904e-26bd73eaea99","Type":"ContainerStarted","Data":"f72c6980c32575092b0735053e6604abe2254dc4764c935f20ab4b36769c3272"}
Nov 03 20:27:10.278640 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:10.277033    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7" event={"ID":"a872c432-f38a-437a-904e-26bd73eaea99","Type":"ContainerStarted","Data":"194a1e8676c7a1e4f8fc87b61eb25a71a8700f857fbd9ab3be89fd70cb090d33"}
Nov 03 20:27:10.278640 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:10.277796    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7"
Nov 03 20:27:10.298264 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:10.298214    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p" podStartSLOduration=2.298200656 podStartE2EDuration="2.298200656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:10.294847454 +0000 UTC m=+1560.427975917" watchObservedRunningTime="2026-11-03 20:27:10.298200656 +0000 UTC m=+1560.431329118"
Nov 03 20:27:10.314130 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:10.314078    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7" podStartSLOduration=2.3140654769999998 podStartE2EDuration="2.314065477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:10.312910224 +0000 UTC m=+1560.446038687" watchObservedRunningTime="2026-11-03 20:27:10.314065477 +0000 UTC m=+1560.447193939"
Nov 03 20:27:11.873546 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:11.873520    4421 scope.go:122] "RemoveContainer" containerID="c81421f21771c03879aabdfdc40c47e1bd18fd88744c2c3d822738cd0ace0e82"
Nov 03 20:27:15.857624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:15.852695    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:27:15.857624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:15.853065    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:27:16.095803 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:16.095744    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:27:16.531625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:16.530693    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:27:16.556625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:16.552945    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:27:16.863679 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:16.852702    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:27:19.645643 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:27:19.640784    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:27:21.312510 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:21.312469    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p"
Nov 03 20:27:22.364043 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:22.362796    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7"
Nov 03 20:27:23.541247 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:23.541213    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w"]
Nov 03 20:27:23.551585 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:23.551523    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w"
Nov 03 20:27:23.556232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:23.556200    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-99a8fe-serving-cert\""
Nov 03 20:27:23.558227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:23.558194    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w"]
Nov 03 20:27:23.635215 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:23.635175    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa825357-78e3-4507-884b-b071ae001d58-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w\" (UID: \"aa825357-78e3-4507-884b-b071ae001d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w"
Nov 03 20:27:23.635399 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:23.635292    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa825357-78e3-4507-884b-b071ae001d58-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w\" (UID: \"aa825357-78e3-4507-884b-b071ae001d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w"
Nov 03 20:27:23.635463 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:23.635422    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqvfs\" (UniqueName: \"kubernetes.io/projected/aa825357-78e3-4507-884b-b071ae001d58-kube-api-access-qqvfs\") pod \"maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w\" (UID: \"aa825357-78e3-4507-884b-b071ae001d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w"
Nov 03 20:27:23.635519 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:23.635505    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa825357-78e3-4507-884b-b071ae001d58-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w\" (UID: \"aa825357-78e3-4507-884b-b071ae001d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w"
Nov 03 20:27:23.739355 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:23.738881    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa825357-78e3-4507-884b-b071ae001d58-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w\" (UID: \"aa825357-78e3-4507-884b-b071ae001d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w"
Nov 03 20:27:23.739355 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:23.739076    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa825357-78e3-4507-884b-b071ae001d58-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w\" (UID: \"aa825357-78e3-4507-884b-b071ae001d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w"
Nov 03 20:27:23.739355 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:23.739146    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa825357-78e3-4507-884b-b071ae001d58-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w\" (UID: \"aa825357-78e3-4507-884b-b071ae001d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w"
Nov 03 20:27:23.739355 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:23.739235    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqvfs\" (UniqueName: \"kubernetes.io/projected/aa825357-78e3-4507-884b-b071ae001d58-kube-api-access-qqvfs\") pod \"maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w\" (UID: \"aa825357-78e3-4507-884b-b071ae001d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w"
Nov 03 20:27:23.740508 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:23.740481    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa825357-78e3-4507-884b-b071ae001d58-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w\" (UID: \"aa825357-78e3-4507-884b-b071ae001d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w"
Nov 03 20:27:23.744448 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:23.744421    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa825357-78e3-4507-884b-b071ae001d58-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w\" (UID: \"aa825357-78e3-4507-884b-b071ae001d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w"
Nov 03 20:27:23.746367 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:23.746340    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa825357-78e3-4507-884b-b071ae001d58-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w\" (UID: \"aa825357-78e3-4507-884b-b071ae001d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w"
Nov 03 20:27:23.755283 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:23.755226    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqvfs\" (UniqueName: \"kubernetes.io/projected/aa825357-78e3-4507-884b-b071ae001d58-kube-api-access-qqvfs\") pod \"maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w\" (UID: \"aa825357-78e3-4507-884b-b071ae001d58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w"
Nov 03 20:27:23.900199 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:23.900099    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w"
Nov 03 20:27:24.572065 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:24.572019    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w"]
Nov 03 20:27:24.578907 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:27:24.578850    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa825357_78e3_4507_884b_b071ae001d58.slice/crio-5f60854a25c95d5b3748a31fe820d77011b6e8758ecf5d79550855462057a311 WatchSource:0}: Error finding container 5f60854a25c95d5b3748a31fe820d77011b6e8758ecf5d79550855462057a311: Status 404 returned error can't find the container with id 5f60854a25c95d5b3748a31fe820d77011b6e8758ecf5d79550855462057a311
Nov 03 20:27:25.603648 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:25.602118    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w" event={"ID":"aa825357-78e3-4507-884b-b071ae001d58","Type":"ContainerStarted","Data":"3d2d8bc0ca86d634f6eabd105e5fd8fcc51440ca8c7ea98943d76f3a627ea46a"}
Nov 03 20:27:25.603648 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:25.602221    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w" event={"ID":"aa825357-78e3-4507-884b-b071ae001d58","Type":"ContainerStarted","Data":"5f60854a25c95d5b3748a31fe820d77011b6e8758ecf5d79550855462057a311"}
Nov 03 20:27:25.603648 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:25.602367    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w"
Nov 03 20:27:25.624220 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:25.624166    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w" podStartSLOduration=1.751532162 podStartE2EDuration="2.624153966s" totalImagesPullingTime="872.621804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:25.623037561 +0000 UTC m=+1575.756166024" watchObservedRunningTime="2026-11-03 20:27:25.624153966 +0000 UTC m=+1575.757282427"
Nov 03 20:27:28.327447 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.327414    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk"]
Nov 03 20:27:28.330804 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.330788    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk"
Nov 03 20:27:28.337011 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.336983    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-6bd876\""
Nov 03 20:27:28.338212 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.338173    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-6bd876-dockercfg-b2pwc\""
Nov 03 20:27:28.342508 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.342484    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk"]
Nov 03 20:27:28.392134 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.392102    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8"]
Nov 03 20:27:28.398308 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.398278    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8"
Nov 03 20:27:28.405096 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.405067    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8"]
Nov 03 20:27:28.536624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.514847    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwsrm\" (UniqueName: \"kubernetes.io/projected/c9c3239a-d59c-4b81-95c6-b89eacc72111-kube-api-access-hwsrm\") pod \"payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk\" (UID: \"c9c3239a-d59c-4b81-95c6-b89eacc72111\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk"
Nov 03 20:27:28.598741 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.586687    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c3239a-d59c-4b81-95c6-b89eacc72111-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk\" (UID: \"c9c3239a-d59c-4b81-95c6-b89eacc72111\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk"
Nov 03 20:27:28.598741 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.586930    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfb2deb-19e8-4407-90f9-d20452299a86-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8\" (UID: \"fcfb2deb-19e8-4407-90f9-d20452299a86\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8"
Nov 03 20:27:28.598741 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.587072    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmgcb\" (UniqueName: \"kubernetes.io/projected/fcfb2deb-19e8-4407-90f9-d20452299a86-kube-api-access-mmgcb\") pod \"payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8\" (UID: \"fcfb2deb-19e8-4407-90f9-d20452299a86\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8"
Nov 03 20:27:28.688333 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.688298    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfb2deb-19e8-4407-90f9-d20452299a86-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8\" (UID: \"fcfb2deb-19e8-4407-90f9-d20452299a86\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8"
Nov 03 20:27:28.688816 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.688792    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmgcb\" (UniqueName: \"kubernetes.io/projected/fcfb2deb-19e8-4407-90f9-d20452299a86-kube-api-access-mmgcb\") pod \"payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8\" (UID: \"fcfb2deb-19e8-4407-90f9-d20452299a86\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8"
Nov 03 20:27:28.693848 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.693822    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwsrm\" (UniqueName: \"kubernetes.io/projected/c9c3239a-d59c-4b81-95c6-b89eacc72111-kube-api-access-hwsrm\") pod \"payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk\" (UID: \"c9c3239a-d59c-4b81-95c6-b89eacc72111\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk"
Nov 03 20:27:28.694075 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.694056    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c3239a-d59c-4b81-95c6-b89eacc72111-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk\" (UID: \"c9c3239a-d59c-4b81-95c6-b89eacc72111\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk"
Nov 03 20:27:28.695247 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.695225    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c3239a-d59c-4b81-95c6-b89eacc72111-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk\" (UID: \"c9c3239a-d59c-4b81-95c6-b89eacc72111\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk"
Nov 03 20:27:28.697944 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.697920    4421 generic.go:356] "Generic (PLEG): container finished" podID="d366ab06-4688-4c4b-b71b-66bb5d20167d" containerID="c6b1e4c79dc0eeb303bea0d33ed7fd995f7f616229ef4c287e6e4f7c5fe371d6" exitCode=0
Nov 03 20:27:28.698257 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.698235    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs" event={"ID":"d366ab06-4688-4c4b-b71b-66bb5d20167d","Type":"ContainerDied","Data":"c6b1e4c79dc0eeb303bea0d33ed7fd995f7f616229ef4c287e6e4f7c5fe371d6"}
Nov 03 20:27:28.698902 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.698885    4421 scope.go:122] "RemoveContainer" containerID="c6b1e4c79dc0eeb303bea0d33ed7fd995f7f616229ef4c287e6e4f7c5fe371d6"
Nov 03 20:27:28.707992 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.707842    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:27:28.708506 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.708482    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:27:28.710039 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.710002    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:27:28.712450 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.712428    4421 generic.go:356] "Generic (PLEG): container finished" podID="4f55c349-f323-4dde-baab-4e279bfcf422" containerID="71a6959fc201244bd7cd969d7edf1cc4c88495e6e053bb2e90440313b33923cf" exitCode=0
Nov 03 20:27:28.727876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.713123    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb" event={"ID":"4f55c349-f323-4dde-baab-4e279bfcf422","Type":"ContainerDied","Data":"71a6959fc201244bd7cd969d7edf1cc4c88495e6e053bb2e90440313b33923cf"}
Nov 03 20:27:28.727876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.713446    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfb2deb-19e8-4407-90f9-d20452299a86-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8\" (UID: \"fcfb2deb-19e8-4407-90f9-d20452299a86\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8"
Nov 03 20:27:28.727876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.726228    4421 scope.go:122] "RemoveContainer" containerID="71a6959fc201244bd7cd969d7edf1cc4c88495e6e053bb2e90440313b33923cf"
Nov 03 20:27:28.733512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.732660    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwsrm\" (UniqueName: \"kubernetes.io/projected/c9c3239a-d59c-4b81-95c6-b89eacc72111-kube-api-access-hwsrm\") pod \"payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk\" (UID: \"c9c3239a-d59c-4b81-95c6-b89eacc72111\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk"
Nov 03 20:27:28.734745 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.734378    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:27:28.734745 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.734482    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:27:28.734745 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.734529    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:27:28.742707 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.742501    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmgcb\" (UniqueName: \"kubernetes.io/projected/fcfb2deb-19e8-4407-90f9-d20452299a86-kube-api-access-mmgcb\") pod \"payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8\" (UID: \"fcfb2deb-19e8-4407-90f9-d20452299a86\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8"
Nov 03 20:27:28.947164 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:28.947133    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk"
Nov 03 20:27:29.029231 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:29.029110    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8"
Nov 03 20:27:29.263076 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:29.261552    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk"]
Nov 03 20:27:29.266455 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:27:29.266397    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c3239a_d59c_4b81_95c6_b89eacc72111.slice/crio-c70bfd415078a9992933d906212d6ba7f988bacce6eb9fa5c3bba6889684f5c8 WatchSource:0}: Error finding container c70bfd415078a9992933d906212d6ba7f988bacce6eb9fa5c3bba6889684f5c8: Status 404 returned error can't find the container with id c70bfd415078a9992933d906212d6ba7f988bacce6eb9fa5c3bba6889684f5c8
Nov 03 20:27:29.421406 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:27:29.421368    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcfb2deb_19e8_4407_90f9_d20452299a86.slice/crio-c2c0cc8992d7d603b713e04fc366e4f6105fa975d18b305a3426782087efe6a3 WatchSource:0}: Error finding container c2c0cc8992d7d603b713e04fc366e4f6105fa975d18b305a3426782087efe6a3: Status 404 returned error can't find the container with id c2c0cc8992d7d603b713e04fc366e4f6105fa975d18b305a3426782087efe6a3
Nov 03 20:27:29.423842 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:29.423452    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8"]
Nov 03 20:27:29.735349 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:29.735316    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8" event={"ID":"fcfb2deb-19e8-4407-90f9-d20452299a86","Type":"ContainerStarted","Data":"c4a98f9811a52e1181f604d28ead48a0b07858f624dff2e83b2fdc4408c8d625"}
Nov 03 20:27:29.735549 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:29.735370    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8" event={"ID":"fcfb2deb-19e8-4407-90f9-d20452299a86","Type":"ContainerStarted","Data":"c2c0cc8992d7d603b713e04fc366e4f6105fa975d18b305a3426782087efe6a3"}
Nov 03 20:27:29.735549 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:29.735514    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8"
Nov 03 20:27:29.737506 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:29.737480    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs" event={"ID":"d366ab06-4688-4c4b-b71b-66bb5d20167d","Type":"ContainerStarted","Data":"56e0455963275decf974ba78a3d7fa578549e25ad0546379e3ac1f0732ccb124"}
Nov 03 20:27:29.739495 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:29.739472    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb" event={"ID":"4f55c349-f323-4dde-baab-4e279bfcf422","Type":"ContainerStarted","Data":"93e84ae9de24da2da7be2109e0b20ddbc51e6d050ad27cb119fe6dc1dc3a7aca"}
Nov 03 20:27:29.745670 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:29.741149    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk" event={"ID":"c9c3239a-d59c-4b81-95c6-b89eacc72111","Type":"ContainerStarted","Data":"d19e26ad6d5a3372c0dc278daee7a8b7c7d6ff78e5fb65026c1f65511787f85b"}
Nov 03 20:27:29.745670 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:29.741190    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk" event={"ID":"c9c3239a-d59c-4b81-95c6-b89eacc72111","Type":"ContainerStarted","Data":"c70bfd415078a9992933d906212d6ba7f988bacce6eb9fa5c3bba6889684f5c8"}
Nov 03 20:27:29.745670 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:29.741787    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk"
Nov 03 20:27:29.762236 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:29.762170    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8" podStartSLOduration=1.762151069 podStartE2EDuration="1.762151069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:29.760013873 +0000 UTC m=+1579.893142337" watchObservedRunningTime="2026-11-03 20:27:29.762151069 +0000 UTC m=+1579.895279532"
Nov 03 20:27:29.851624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:29.842243    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk" podStartSLOduration=1.8422246279999999 podStartE2EDuration="1.842224628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:29.838723718 +0000 UTC m=+1579.971852180" watchObservedRunningTime="2026-11-03 20:27:29.842224628 +0000 UTC m=+1579.975353093"
Nov 03 20:27:29.875356 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:29.870488    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:27:29.875683 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:29.870698    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 03 20:27:29.875858 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:29.875835    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs" podUID="d366ab06-4688-4c4b-b71b-66bb5d20167d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 03 20:27:30.555621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:30.551924    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:27:30.578621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:30.561786    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Nov 03 20:27:30.578621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:30.561844    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb" podUID="4f55c349-f323-4dde-baab-4e279bfcf422" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Nov 03 20:27:30.864868 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:30.864748    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 03 20:27:30.865154 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:30.865124    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs" podUID="d366ab06-4688-4c4b-b71b-66bb5d20167d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 03 20:27:31.543126 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.543090    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Nov 03 20:27:31.543479 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.543441    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb" podUID="4f55c349-f323-4dde-baab-4e279bfcf422" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Nov 03 20:27:31.618972 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.617160    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"]
Nov 03 20:27:31.631712 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.631677    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w"
Nov 03 20:27:31.633009 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.632982    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:31.633651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.633625    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"]
Nov 03 20:27:31.729403 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.729372    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-3c25bf-openshift-default-dockercfg-bmp7d\""
Nov 03 20:27:31.740375 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.740341    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e840e124-446a-4ef7-8ed3-abc704048d25-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:31.740738 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.740713    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:31.791854 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.785209    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:31.792728 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.792700    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:31.792944 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.792922    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e840e124-446a-4ef7-8ed3-abc704048d25-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:31.793096 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.793078    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:31.793330 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.793276    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e840e124-446a-4ef7-8ed3-abc704048d25-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:31.793574 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.793554    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:31.793855 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.793834    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6tjw\" (UniqueName: \"kubernetes.io/projected/e840e124-446a-4ef7-8ed3-abc704048d25-kube-api-access-q6tjw\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:31.907675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.907629    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:31.907873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.907740    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:31.907873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.907796    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:31.907873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.907855    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e840e124-446a-4ef7-8ed3-abc704048d25-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:31.908113 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.907900    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:31.908113 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.908001    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e840e124-446a-4ef7-8ed3-abc704048d25-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:31.908113 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.908108    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:31.908265 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.908212    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6tjw\" (UniqueName: \"kubernetes.io/projected/e840e124-446a-4ef7-8ed3-abc704048d25-kube-api-access-q6tjw\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:31.908265 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.908256    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e840e124-446a-4ef7-8ed3-abc704048d25-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:31.909108 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.909000    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e840e124-446a-4ef7-8ed3-abc704048d25-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:31.909544 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.909522    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:31.909810 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.909793    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:31.910021 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.910005    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:31.911654 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.910823    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 03 20:27:31.911654 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.910881    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs" podUID="d366ab06-4688-4c4b-b71b-66bb5d20167d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 03 20:27:31.911654 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.911038    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:31.914836 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.914814    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:31.927946 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.922935    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e840e124-446a-4ef7-8ed3-abc704048d25-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:31.927946 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.925277    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e840e124-446a-4ef7-8ed3-abc704048d25-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:31.928731 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:31.928439    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6tjw\" (UniqueName: \"kubernetes.io/projected/e840e124-446a-4ef7-8ed3-abc704048d25-kube-api-access-q6tjw\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:32.017367 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:32.017184    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:32.608377 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:32.547688    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Nov 03 20:27:32.608692 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:32.608659    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb" podUID="4f55c349-f323-4dde-baab-4e279bfcf422" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Nov 03 20:27:32.610659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:32.610583    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"]
Nov 03 20:27:32.613115 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:27:32.613083    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode840e124_446a_4ef7_8ed3_abc704048d25.slice/crio-8bf9bf356ea121afe51f521c77d73d1acd1bf29909823dbf10d49cc24f80b91e WatchSource:0}: Error finding container 8bf9bf356ea121afe51f521c77d73d1acd1bf29909823dbf10d49cc24f80b91e: Status 404 returned error can't find the container with id 8bf9bf356ea121afe51f521c77d73d1acd1bf29909823dbf10d49cc24f80b91e
Nov 03 20:27:32.619420 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:32.618375    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:27:32.619420 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:32.618457    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:27:32.619420 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:32.618501    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:27:32.678852 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:32.678819    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:27:32.680592 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:32.680560    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:27:32.829630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:32.829539    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw" event={"ID":"e840e124-446a-4ef7-8ed3-abc704048d25","Type":"ContainerStarted","Data":"a545fe0897cffeef80b749e593bc1323b971d280fda0d0709373c4ceaa0bd31c"}
Nov 03 20:27:32.829630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:32.829611    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw" event={"ID":"e840e124-446a-4ef7-8ed3-abc704048d25","Type":"ContainerStarted","Data":"8bf9bf356ea121afe51f521c77d73d1acd1bf29909823dbf10d49cc24f80b91e"}
Nov 03 20:27:32.879038 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:32.879000    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 03 20:27:32.879306 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:32.879280    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs" podUID="d366ab06-4688-4c4b-b71b-66bb5d20167d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 03 20:27:33.026655 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.019832    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:33.026655 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.023006    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 03 20:27:33.026655 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.023117    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw" podUID="e840e124-446a-4ef7-8ed3-abc704048d25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 03 20:27:33.066631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.060975    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw" podStartSLOduration=2.0609515959999998 podStartE2EDuration="2.060951596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:32.87458618 +0000 UTC m=+1583.007714656" watchObservedRunningTime="2026-11-03 20:27:33.060951596 +0000 UTC m=+1583.194080060"
Nov 03 20:27:33.066631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.064511    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"]
Nov 03 20:27:33.070632 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.069707    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.095673 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.094144    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-3211a0-openshift-default-dockercfg-57hrh\""
Nov 03 20:27:33.116627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.114819    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f788\" (UniqueName: \"kubernetes.io/projected/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-kube-api-access-4f788\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.116627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.114874    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.116627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.114952    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.116627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.114995    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.116627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.115076    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.116627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.115220    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.116627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.115398    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.116627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.115426    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.116627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.115503    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.152827 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.148652    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"]
Nov 03 20:27:33.192448 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.192397    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"]
Nov 03 20:27:33.192868 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.192852    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.231049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.219214    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.231049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.219276    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.231049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.219310    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz795\" (UniqueName: \"kubernetes.io/projected/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-kube-api-access-hz795\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.231049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.219367    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.231049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.219454    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.231049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.219488    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.231049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.219546    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.231049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.219578    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.231049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.219707    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.260950 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.241633    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.260950 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.241913    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.261846    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.262083    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.262135    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.262178    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.262227    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.262282    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.262353    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f788\" (UniqueName: \"kubernetes.io/projected/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-kube-api-access-4f788\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.262390    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.262423    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.262466    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.262996    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.263712    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.327070    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-bab2c3-openshift-default-dockercfg-bkx62\""
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.328460    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"]
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.369695    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.369757    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.369806    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.369894    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.369937    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.370032    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz795\" (UniqueName: \"kubernetes.io/projected/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-kube-api-access-hz795\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.370191    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.370271    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.370305    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.370887    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.372350    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.372910    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.373397    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.373743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.373653    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.418064 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.417560    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f788\" (UniqueName: \"kubernetes.io/projected/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-kube-api-access-4f788\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.425962 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.425049    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.425962 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.425435    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.429478 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.427514    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.429478 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.428478    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.429478 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.429031    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz795\" (UniqueName: \"kubernetes.io/projected/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-kube-api-access-hz795\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.429782 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.429480    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.429782 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.429545    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.454465 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.453670    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:33.508887 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.508849    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:33.739791 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.738860    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Nov 03 20:27:33.739791 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.738920    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb" podUID="4f55c349-f323-4dde-baab-4e279bfcf422" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Nov 03 20:27:33.921624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.918791    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 03 20:27:33.921624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:33.918865    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs" podUID="d366ab06-4688-4c4b-b71b-66bb5d20167d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 03 20:27:34.030928 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:34.029076    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 03 20:27:34.030928 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:34.029138    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw" podUID="e840e124-446a-4ef7-8ed3-abc704048d25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 03 20:27:35.062624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:35.042170    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:27:35.062624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:35.042725    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:27:35.062624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:35.042989    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:27:35.197674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:35.188024    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 03 20:27:35.197674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:35.188107    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw" podUID="e840e124-446a-4ef7-8ed3-abc704048d25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 03 20:27:35.323625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:35.303968    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 03 20:27:35.323625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:35.304039    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs" podUID="d366ab06-4688-4c4b-b71b-66bb5d20167d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 03 20:27:35.366621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:35.363017    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Nov 03 20:27:35.366621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:35.363131    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb" podUID="4f55c349-f323-4dde-baab-4e279bfcf422" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Nov 03 20:27:35.637347 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:35.637245    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:27:35.637625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:35.637585    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:27:35.637835 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:35.637815    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:27:35.653017 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:35.648963    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body=
Nov 03 20:27:35.653017 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:35.649012    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb" podUID="4f55c349-f323-4dde-baab-4e279bfcf422" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused"
Nov 03 20:27:35.692622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:35.681214    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"]
Nov 03 20:27:35.738708 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:35.738334    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"]
Nov 03 20:27:35.832633 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:35.821026    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz" event={"ID":"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a","Type":"ContainerStarted","Data":"322975a6794dbfea552bc5849be2d6b5afcd931f58c86a73893d4846db921715"}
Nov 03 20:27:35.832633 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:35.825362    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7" event={"ID":"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511","Type":"ContainerStarted","Data":"68c8ef004c764f91684ccdd253872c2feb8a540a9772b92c440dc2d826abddb2"}
Nov 03 20:27:35.883352 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:35.867936    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 03 20:27:35.883352 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:35.867994    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs" podUID="d366ab06-4688-4c4b-b71b-66bb5d20167d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 03 20:27:36.018124 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:36.018073    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 03 20:27:36.018331 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:36.018128    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw" podUID="e840e124-446a-4ef7-8ed3-abc704048d25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 03 20:27:36.886214 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:36.886129    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7" event={"ID":"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511","Type":"ContainerStarted","Data":"a16db7da3590f2b747019ce5b5365908f90124e9ec3f444441cf374abe373047"}
Nov 03 20:27:36.888285 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:36.888250    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:27:36.890627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:36.890612    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 20:27:36.893529 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:36.891792    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz" event={"ID":"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a","Type":"ContainerStarted","Data":"d04db38f84c76802ee960c31b7f975c6f0af370de4de81443983e2c554fea558"}
Nov 03 20:27:36.914624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:36.912485    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7" podStartSLOduration=3.912470482 podStartE2EDuration="3.912470482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:36.911118161 +0000 UTC m=+1587.044246624" watchObservedRunningTime="2026-11-03 20:27:36.912470482 +0000 UTC m=+1587.045598935"
Nov 03 20:27:36.977543 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:36.971486    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:27:36.998631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:36.979972    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 20:27:37.017073 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:37.017006    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz" podStartSLOduration=4.016952761 podStartE2EDuration="4.016952761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:37.010834603 +0000 UTC m=+1587.143963065" watchObservedRunningTime="2026-11-03 20:27:37.016952761 +0000 UTC m=+1587.150081227"
Nov 03 20:27:37.023820 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:37.023784    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 03 20:27:37.024082 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:37.024053    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw" podUID="e840e124-446a-4ef7-8ed3-abc704048d25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 03 20:27:37.502690 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:37.496054    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:37.537625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:37.529162    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:37.554625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:37.545681    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 03 20:27:37.554625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:37.545779    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7" podUID="2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 03 20:27:37.583740 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:37.581737    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 03 20:27:37.583740 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:37.581809    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz" podUID="873d9cbd-c45e-4c14-90bb-e0ca79c56e7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 03 20:27:38.018622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:38.017635    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 03 20:27:38.018622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:38.017707    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw" podUID="e840e124-446a-4ef7-8ed3-abc704048d25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 03 20:27:38.457992 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:38.457954    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 03 20:27:38.458268 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:38.458241    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7" podUID="2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 03 20:27:38.524625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:38.513094    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 03 20:27:38.524625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:38.516657    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz" podUID="873d9cbd-c45e-4c14-90bb-e0ca79c56e7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 03 20:27:39.109835 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:39.109792    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 03 20:27:39.110333 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:39.109858    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw" podUID="e840e124-446a-4ef7-8ed3-abc704048d25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 03 20:27:39.461611 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:39.461553    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 03 20:27:39.472857 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:39.472817    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7" podUID="2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 03 20:27:39.531713 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:39.524447    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 03 20:27:39.531713 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:39.524534    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz" podUID="873d9cbd-c45e-4c14-90bb-e0ca79c56e7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 03 20:27:40.057485 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:40.057039    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body=
Nov 03 20:27:40.057485 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:40.057100    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw" podUID="e840e124-446a-4ef7-8ed3-abc704048d25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused"
Nov 03 20:27:40.459632 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:40.458743    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 03 20:27:40.459632 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:40.458807    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7" podUID="2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 03 20:27:40.515365 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:40.515332    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 03 20:27:40.515665 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:40.515591    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz" podUID="873d9cbd-c45e-4c14-90bb-e0ca79c56e7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 03 20:27:40.793727 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:40.793636    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8"
Nov 03 20:27:41.022805 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:41.022770    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:41.023272 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:41.023249    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:41.024033 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:41.024011    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:27:41.462481 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:41.462448    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 03 20:27:41.463102 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:41.463070    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7" podUID="2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 03 20:27:41.509877 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:41.509850    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 03 20:27:41.510129 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:41.510104    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz" podUID="873d9cbd-c45e-4c14-90bb-e0ca79c56e7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 03 20:27:41.821234 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:41.821149    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk"
Nov 03 20:27:42.464373 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:42.464288    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 03 20:27:42.499502 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:42.499397    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7" podUID="2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 03 20:27:42.512500 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:42.512421    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 03 20:27:42.512806 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:42.512779    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz" podUID="873d9cbd-c45e-4c14-90bb-e0ca79c56e7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 03 20:27:43.456061 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:43.456026    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 03 20:27:43.456286 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:43.456081    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7" podUID="2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 03 20:27:43.514566 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:43.514535    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:43.515088 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:43.515075    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:43.515905 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:43.515884    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:27:44.459765 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:44.459738    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:44.460219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:44.460206    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:44.461000 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:27:44.460984    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:27:49.629901 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:27:49.629868    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:28:00.450200 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:00.450166    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"]
Nov 03 20:28:00.450759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:00.450398    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m" podUID="2982ead4-1f06-4ce4-af76-81e660252c20" containerName="istio-proxy" containerID="cri-o://f17136f14f8b01b334fef683cecfcc87229538154de7a09e459c547d72cd3c02" gracePeriod=30
Nov 03 20:28:05.774893 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.774864    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:28:05.804230 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.804194    4421 generic.go:356] "Generic (PLEG): container finished" podID="2982ead4-1f06-4ce4-af76-81e660252c20" containerID="f17136f14f8b01b334fef683cecfcc87229538154de7a09e459c547d72cd3c02" exitCode=0
Nov 03 20:28:05.804449 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.804426    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m" event={"ID":"2982ead4-1f06-4ce4-af76-81e660252c20","Type":"ContainerDied","Data":"f17136f14f8b01b334fef683cecfcc87229538154de7a09e459c547d72cd3c02"}
Nov 03 20:28:05.804519 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.804491    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m" event={"ID":"2982ead4-1f06-4ce4-af76-81e660252c20","Type":"ContainerDied","Data":"2319eed735a3107a684ed15ea5b2c9e45e11137e40c2348c386384bdb7fe26ed"}
Nov 03 20:28:05.804572 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.804522    4421 scope.go:122] "RemoveContainer" containerID="f17136f14f8b01b334fef683cecfcc87229538154de7a09e459c547d72cd3c02"
Nov 03 20:28:05.804768 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.804718    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"
Nov 03 20:28:05.822311 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.822229    4421 scope.go:122] "RemoveContainer" containerID="f17136f14f8b01b334fef683cecfcc87229538154de7a09e459c547d72cd3c02"
Nov 03 20:28:05.822965 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:28:05.822855    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f17136f14f8b01b334fef683cecfcc87229538154de7a09e459c547d72cd3c02\": container with ID starting with f17136f14f8b01b334fef683cecfcc87229538154de7a09e459c547d72cd3c02 not found: ID does not exist" containerID="f17136f14f8b01b334fef683cecfcc87229538154de7a09e459c547d72cd3c02"
Nov 03 20:28:05.822965 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.822911    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f17136f14f8b01b334fef683cecfcc87229538154de7a09e459c547d72cd3c02"} err="failed to get container status \"f17136f14f8b01b334fef683cecfcc87229538154de7a09e459c547d72cd3c02\": rpc error: code = NotFound desc = could not find container \"f17136f14f8b01b334fef683cecfcc87229538154de7a09e459c547d72cd3c02\": container with ID starting with f17136f14f8b01b334fef683cecfcc87229538154de7a09e459c547d72cd3c02 not found: ID does not exist"
Nov 03 20:28:05.870899 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.870864    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-workload-socket\") pod \"2982ead4-1f06-4ce4-af76-81e660252c20\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") "
Nov 03 20:28:05.871085 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.870950    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2982ead4-1f06-4ce4-af76-81e660252c20-istio-token\" (UniqueName: \"kubernetes.io/projected/2982ead4-1f06-4ce4-af76-81e660252c20-istio-token\") pod \"2982ead4-1f06-4ce4-af76-81e660252c20\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") "
Nov 03 20:28:05.871085 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.870988    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-workload-certs\") pod \"2982ead4-1f06-4ce4-af76-81e660252c20\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") "
Nov 03 20:28:05.871085 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.871024    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2982ead4-1f06-4ce4-af76-81e660252c20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2982ead4-1f06-4ce4-af76-81e660252c20-istiod-ca-cert\") pod \"2982ead4-1f06-4ce4-af76-81e660252c20\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") "
Nov 03 20:28:05.871265 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.871114    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2982ead4-1f06-4ce4-af76-81e660252c20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2982ead4-1f06-4ce4-af76-81e660252c20-istio-podinfo\") pod \"2982ead4-1f06-4ce4-af76-81e660252c20\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") "
Nov 03 20:28:05.871265 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.871163    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-workload-socket" pod "2982ead4-1f06-4ce4-af76-81e660252c20" (UID: "2982ead4-1f06-4ce4-af76-81e660252c20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:28:05.871265 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.871210    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-istio-envoy\") pod \"2982ead4-1f06-4ce4-af76-81e660252c20\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") "
Nov 03 20:28:05.871438 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.871267    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-credential-socket\") pod \"2982ead4-1f06-4ce4-af76-81e660252c20\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") "
Nov 03 20:28:05.871438 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.871307    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2982ead4-1f06-4ce4-af76-81e660252c20-kube-api-access-dx6pz\" (UniqueName: \"kubernetes.io/projected/2982ead4-1f06-4ce4-af76-81e660252c20-kube-api-access-dx6pz\") pod \"2982ead4-1f06-4ce4-af76-81e660252c20\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") "
Nov 03 20:28:05.871438 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.871389    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-workload-certs" pod "2982ead4-1f06-4ce4-af76-81e660252c20" (UID: "2982ead4-1f06-4ce4-af76-81e660252c20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:28:05.872729 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.871687    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-istio-data\") pod \"2982ead4-1f06-4ce4-af76-81e660252c20\" (UID: \"2982ead4-1f06-4ce4-af76-81e660252c20\") "
Nov 03 20:28:05.872729 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.871717    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2982ead4-1f06-4ce4-af76-81e660252c20-istiod-ca-cert" pod "2982ead4-1f06-4ce4-af76-81e660252c20" (UID: "2982ead4-1f06-4ce4-af76-81e660252c20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:28:05.872729 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.871981    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-istio-data" pod "2982ead4-1f06-4ce4-af76-81e660252c20" (UID: "2982ead4-1f06-4ce4-af76-81e660252c20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:28:05.872729 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.872267    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-credential-socket" pod "2982ead4-1f06-4ce4-af76-81e660252c20" (UID: "2982ead4-1f06-4ce4-af76-81e660252c20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:28:05.872729 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.872668    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:05.872729 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.872683    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:05.872729 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.872693    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:05.872729 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.872701    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2982ead4-1f06-4ce4-af76-81e660252c20-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:05.872729 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.872709    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:05.877114 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.877088    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2982ead4-1f06-4ce4-af76-81e660252c20-kube-api-access-dx6pz" pod "2982ead4-1f06-4ce4-af76-81e660252c20" (UID: "2982ead4-1f06-4ce4-af76-81e660252c20"). InnerVolumeSpecName "kube-api-access-dx6pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:28:05.878095 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.878061    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-istio-envoy" pod "2982ead4-1f06-4ce4-af76-81e660252c20" (UID: "2982ead4-1f06-4ce4-af76-81e660252c20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:28:05.882681 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.882655    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2982ead4-1f06-4ce4-af76-81e660252c20-istio-podinfo" pod "2982ead4-1f06-4ce4-af76-81e660252c20" (UID: "2982ead4-1f06-4ce4-af76-81e660252c20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:28:05.884810 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.884774    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2982ead4-1f06-4ce4-af76-81e660252c20-istio-token" pod "2982ead4-1f06-4ce4-af76-81e660252c20" (UID: "2982ead4-1f06-4ce4-af76-81e660252c20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:28:05.974913 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.974839    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2982ead4-1f06-4ce4-af76-81e660252c20-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:05.974913 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.974870    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2982ead4-1f06-4ce4-af76-81e660252c20-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:05.974913 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.974882    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2982ead4-1f06-4ce4-af76-81e660252c20-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:05.974913 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:05.974891    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx6pz\" (UniqueName: \"kubernetes.io/projected/2982ead4-1f06-4ce4-af76-81e660252c20-kube-api-access-dx6pz\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:06.129712 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:06.129664    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"]
Nov 03 20:28:06.136376 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:06.136345    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-gvh7m"]
Nov 03 20:28:08.017823 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:08.017782    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2982ead4-1f06-4ce4-af76-81e660252c20" path="/var/lib/kubelet/pods/2982ead4-1f06-4ce4-af76-81e660252c20/volumes"
Nov 03 20:28:16.900494 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:16.900464    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"]
Nov 03 20:28:16.901266 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:16.901250    4421 cpu_manager.go:414] "containerMap: removing container" podUID="2982ead4-1f06-4ce4-af76-81e660252c20" containerName="istio-proxy"
Nov 03 20:28:16.901266 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:16.901267    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2982ead4-1f06-4ce4-af76-81e660252c20" containerName="istio-proxy"
Nov 03 20:28:16.901421 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:16.901411    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="2982ead4-1f06-4ce4-af76-81e660252c20" containerName="istio-proxy"
Nov 03 20:28:16.906917 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:16.906898    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:16.922750 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:16.922692    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"]
Nov 03 20:28:16.955816 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:16.955784    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"]
Nov 03 20:28:16.961642 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:16.961577    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:16.968923 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:16.968892    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"]
Nov 03 20:28:17.057941 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.057904    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbxqb\" (UniqueName: \"kubernetes.io/projected/5a14b4da-8ce7-457e-89b8-6e4788d924f0-kube-api-access-sbxqb\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.058109 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.057946    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txswl\" (UniqueName: \"kubernetes.io/projected/3d2b0564-e201-4bab-9dec-422ecc7955b1-kube-api-access-txswl\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.058109 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.058032    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.058185 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.058120    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.058237 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.058223    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.058273 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.058255    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.058355 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.058332    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.058471 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.058456    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.058520 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.058507    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.058579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.058564    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2b0564-e201-4bab-9dec-422ecc7955b1-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.058730 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.058714    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.058792 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.058771    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.058936 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.058851    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.058936 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.058891    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.059038 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.058970    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.059038 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.059006    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.059147 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.059051    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.059147 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.059094    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.160479 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.160398    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.160479 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.160441    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.160745 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.160485    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbxqb\" (UniqueName: \"kubernetes.io/projected/5a14b4da-8ce7-457e-89b8-6e4788d924f0-kube-api-access-sbxqb\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.160745 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.160513    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txswl\" (UniqueName: \"kubernetes.io/projected/3d2b0564-e201-4bab-9dec-422ecc7955b1-kube-api-access-txswl\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.160745 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.160552    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.160745 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.160576    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.160745 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.160660    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.160745 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.160698    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.160745 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.160732    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.161124 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.160829    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.161124 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.160862    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.161124 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.160925    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2b0564-e201-4bab-9dec-422ecc7955b1-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.161124 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.160960    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.161124 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.161016    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.161124 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.161065    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.161451 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.161185    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.161451 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.161237    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.161451 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.161271    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.161451 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.161345    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.161451 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.161406    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.161451 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.161444    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.161765 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.161489    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.161765 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.161526    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.161765 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.161542    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.161765 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.161488    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.161765 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.161748    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2b0564-e201-4bab-9dec-422ecc7955b1-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.162041 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.162021    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.162129 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.162107    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.163415 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.163394    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.163666 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.163646    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.164262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.164239    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.164478 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.164462    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.173351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.173327    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txswl\" (UniqueName: \"kubernetes.io/projected/3d2b0564-e201-4bab-9dec-422ecc7955b1-kube-api-access-txswl\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.173532 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.173512    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbxqb\" (UniqueName: \"kubernetes.io/projected/5a14b4da-8ce7-457e-89b8-6e4788d924f0-kube-api-access-sbxqb\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.174246 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.174224    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.174667 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.174647    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.228040 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.228010    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:17.282949 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.282918    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:17.526150 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.526117    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"]
Nov 03 20:28:17.526656 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:28:17.526626    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2b0564_e201_4bab_9dec_422ecc7955b1.slice/crio-d9c68783fd4b11d78f2e0b3cb415d08cd6781f168e7e82d2058921e16112c4ed WatchSource:0}: Error finding container d9c68783fd4b11d78f2e0b3cb415d08cd6781f168e7e82d2058921e16112c4ed: Status 404 returned error can't find the container with id d9c68783fd4b11d78f2e0b3cb415d08cd6781f168e7e82d2058921e16112c4ed
Nov 03 20:28:17.529638 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.529578    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:28:17.529745 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.529675    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:28:17.529745 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.529710    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:28:17.683238 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.683204    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"]
Nov 03 20:28:17.683960 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:28:17.683932    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a14b4da_8ce7_457e_89b8_6e4788d924f0.slice/crio-8e396394a94d1304afb2a5c38f5b7c304987fddaffcf3abf09493957728c2d83 WatchSource:0}: Error finding container 8e396394a94d1304afb2a5c38f5b7c304987fddaffcf3abf09493957728c2d83: Status 404 returned error can't find the container with id 8e396394a94d1304afb2a5c38f5b7c304987fddaffcf3abf09493957728c2d83
Nov 03 20:28:17.686971 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.686939    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:28:17.687057 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.687012    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:28:17.687057 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.687041    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:28:17.918949 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.918917    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4" event={"ID":"3d2b0564-e201-4bab-9dec-422ecc7955b1","Type":"ContainerStarted","Data":"ac70a7a5a44774198c2eaf131db35305d9d0f77dfefb7ebf50f57a271607ab65"}
Nov 03 20:28:17.919426 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.918977    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4" event={"ID":"3d2b0564-e201-4bab-9dec-422ecc7955b1","Type":"ContainerStarted","Data":"d9c68783fd4b11d78f2e0b3cb415d08cd6781f168e7e82d2058921e16112c4ed"}
Nov 03 20:28:17.920849 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.920821    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt" event={"ID":"5a14b4da-8ce7-457e-89b8-6e4788d924f0","Type":"ContainerStarted","Data":"ec9d322207b181bfaba3248655446b57cc3138cd2dd31cd41e0cbe3b529d6eff"}
Nov 03 20:28:17.920979 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.920871    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt" event={"ID":"5a14b4da-8ce7-457e-89b8-6e4788d924f0","Type":"ContainerStarted","Data":"8e396394a94d1304afb2a5c38f5b7c304987fddaffcf3abf09493957728c2d83"}
Nov 03 20:28:17.944551 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.944484    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4" podStartSLOduration=1.944466508 podStartE2EDuration="1.944466508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:17.940796446 +0000 UTC m=+1628.073924936" watchObservedRunningTime="2026-11-03 20:28:17.944466508 +0000 UTC m=+1628.077594970"
Nov 03 20:28:17.979854 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:17.977845    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt" podStartSLOduration=1.977826192 podStartE2EDuration="1.977826192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:17.96278792 +0000 UTC m=+1628.095916410" watchObservedRunningTime="2026-11-03 20:28:17.977826192 +0000 UTC m=+1628.110954658"
Nov 03 20:28:18.227985 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.227950    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"]
Nov 03 20:28:18.236722 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.235938    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 03 20:28:18.236722 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.236001    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4" podUID="3d2b0564-e201-4bab-9dec-422ecc7955b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 03 20:28:18.266478 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.266446    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:18.266906 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.266886    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"]
Nov 03 20:28:18.267798 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.266826    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.291172 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.291146    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:18.291620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.291585    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"]
Nov 03 20:28:18.291886 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.291870    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"]
Nov 03 20:28:18.293696 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.291463    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.301278 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.300971    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:28:18.301278 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.301023    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt" podUID="5a14b4da-8ce7-457e-89b8-6e4788d924f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 03 20:28:18.318630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.315046    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"]
Nov 03 20:28:18.318630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.315102    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"]
Nov 03 20:28:18.318630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.315205    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.329907 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.326674    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4mz9\" (UniqueName: \"kubernetes.io/projected/29af63ab-34a3-4ee3-9a74-b8108503642e-kube-api-access-n4mz9\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.329907 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.326757    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.329907 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.326790    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.329907 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.326831    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.329907 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.326861    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.329907 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.326990    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.329907 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.327025    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.329907 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.327069    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.329907 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.327102    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.330786 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.330671    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.330786 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.330742    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.338341 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.338024    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.338341 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.338061    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.338341 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.338100    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.338341 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.338127    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.338341 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.338155    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.338341 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.338182    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.339055 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.338802    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.339055 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.338843    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29af63ab-34a3-4ee3-9a74-b8108503642e-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.339055 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.338951    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.339227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.339061    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.339227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.339134    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjkv5\" (UniqueName: \"kubernetes.io/projected/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-kube-api-access-tjkv5\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.339227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.339166    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.339227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.339189    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.339441 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.339224    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzt7p\" (UniqueName: \"kubernetes.io/projected/ad241f3c-90c0-4f0f-b90c-47eec0af0245-kube-api-access-gzt7p\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.339441 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.339249    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.339441 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.339275    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.460829 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.460785    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4mz9\" (UniqueName: \"kubernetes.io/projected/29af63ab-34a3-4ee3-9a74-b8108503642e-kube-api-access-n4mz9\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.461544 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.461517    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.470732 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.470698    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.471016 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.470994    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.474858 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.474834    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.475159 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.475142    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.475264 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.475253    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.475348 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.475340    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.475422 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.475413    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.475504 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.475495    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.475570 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.475561    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.475770 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.475752    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.475896 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.475878    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.476028 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.476013    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.476150 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.476135    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.476267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.476253    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.476347 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.466227    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.476452 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.476434    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.476577 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.476561    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.476764 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.476746    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29af63ab-34a3-4ee3-9a74-b8108503642e-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.476874 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.476854    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.476969 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.476954    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.477140 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.477125    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.477308 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.477291    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjkv5\" (UniqueName: \"kubernetes.io/projected/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-kube-api-access-tjkv5\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.477425 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.477409    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.477534 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.477519    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.477680 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.477663    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzt7p\" (UniqueName: \"kubernetes.io/projected/ad241f3c-90c0-4f0f-b90c-47eec0af0245-kube-api-access-gzt7p\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.477812 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.477793    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.477937 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.477919    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.478172 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.478153    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29af63ab-34a3-4ee3-9a74-b8108503642e-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.478246 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.476612    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.478246 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.476019    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.478364 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.473005    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.478364 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.476323    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.480940 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.479180    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.480940 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.480147    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.480940 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.480369    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.481433 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.481406    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.481827 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.481805    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.482553 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.482531    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.484555 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.484325    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.484555 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.484510    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.493567 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.493160    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.501681 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.501572    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.502325 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.502271    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.502819 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.502767    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.503673 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.503548    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4mz9\" (UniqueName: \"kubernetes.io/projected/29af63ab-34a3-4ee3-9a74-b8108503642e-kube-api-access-n4mz9\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.512971 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.512848    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.513622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.513527    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.514050 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.514029    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.517076 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.517032    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.519076 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.518487    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:18.519076 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.519031    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzt7p\" (UniqueName: \"kubernetes.io/projected/ad241f3c-90c0-4f0f-b90c-47eec0af0245-kube-api-access-gzt7p\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.519282 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.519252    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjkv5\" (UniqueName: \"kubernetes.io/projected/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-kube-api-access-tjkv5\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.688109 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.688006    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:18.708746 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.708715    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:18.722122 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:18.721719    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:19.228507 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:19.228471    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 03 20:28:19.228897 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:19.228534    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4" podUID="3d2b0564-e201-4bab-9dec-422ecc7955b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 03 20:28:19.300436 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:19.300020    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:28:19.300436 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:19.300076    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt" podUID="5a14b4da-8ce7-457e-89b8-6e4788d924f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 03 20:28:19.331872 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:19.326155    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:28:19.331872 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:19.326336    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:28:19.331872 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:19.326473    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:28:19.331872 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:19.327166    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"]
Nov 03 20:28:19.521637 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:19.521567    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"]
Nov 03 20:28:19.525507 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:19.525438    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:28:19.525591 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:19.525508    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:28:19.525591 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:19.525539    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:28:19.635986 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:19.635758    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"]
Nov 03 20:28:19.646998 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:28:19.646942    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29af63ab_34a3_4ee3_9a74_b8108503642e.slice/crio-33c6d14bfe9fb032bcb33cb52ddbd86a80335815835f51ab33e0f63e1ca7785c WatchSource:0}: Error finding container 33c6d14bfe9fb032bcb33cb52ddbd86a80335815835f51ab33e0f63e1ca7785c: Status 404 returned error can't find the container with id 33c6d14bfe9fb032bcb33cb52ddbd86a80335815835f51ab33e0f63e1ca7785c
Nov 03 20:28:19.659499 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:19.659079    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:28:19.659499 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:19.659176    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:28:19.659499 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:19.659217    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:28:19.699157 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:28:19.699040    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:28:20.368623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:20.361512    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6" event={"ID":"ad241f3c-90c0-4f0f-b90c-47eec0af0245","Type":"ContainerStarted","Data":"ed0605f07816dc22b4488f407845fe8554875e6c77e081ad47f800edb27ac235"}
Nov 03 20:28:20.368623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:20.361575    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6" event={"ID":"ad241f3c-90c0-4f0f-b90c-47eec0af0245","Type":"ContainerStarted","Data":"89eb04cdf6f1b2552d34e8e41df6b96bd3936bb8295c6f40a6c4576753237cd5"}
Nov 03 20:28:20.398135 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:20.392817    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:28:20.398135 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:20.392875    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt" podUID="5a14b4da-8ce7-457e-89b8-6e4788d924f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 03 20:28:20.398135 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:20.397942    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 03 20:28:20.398135 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:20.397990    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4" podUID="3d2b0564-e201-4bab-9dec-422ecc7955b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 03 20:28:20.409892 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:20.409856    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4" event={"ID":"29af63ab-34a3-4ee3-9a74-b8108503642e","Type":"ContainerStarted","Data":"33c6d14bfe9fb032bcb33cb52ddbd86a80335815835f51ab33e0f63e1ca7785c"}
Nov 03 20:28:20.432693 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:20.431904    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6" podStartSLOduration=2.431890183 podStartE2EDuration="2.431890183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:20.421175513 +0000 UTC m=+1630.554303975" watchObservedRunningTime="2026-11-03 20:28:20.431890183 +0000 UTC m=+1630.565018645"
Nov 03 20:28:20.438337 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:20.438300    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69" event={"ID":"d6bfef45-f4d7-4482-bde4-2328d5e54b6c","Type":"ContainerStarted","Data":"92068db831ba35380885d023f4660762f11da1800947f47c0915dcac8035ba5b"}
Nov 03 20:28:20.480290 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:20.462035    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4" podStartSLOduration=2.462014621 podStartE2EDuration="2.462014621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:20.460458802 +0000 UTC m=+1630.593587266" watchObservedRunningTime="2026-11-03 20:28:20.462014621 +0000 UTC m=+1630.595143083"
Nov 03 20:28:20.654632 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:20.642568    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69" podStartSLOduration=2.642545723 podStartE2EDuration="2.642545723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:20.552668557 +0000 UTC m=+1630.685797020" watchObservedRunningTime="2026-11-03 20:28:20.642545723 +0000 UTC m=+1630.775674187"
Nov 03 20:28:20.702687 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:20.699188    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:20.702687 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:20.699364    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:28:20.702687 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:20.699420    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6" podUID="ad241f3c-90c0-4f0f-b90c-47eec0af0245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 03 20:28:20.732170 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:20.732135    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:20.732520 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:20.732488    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:21.243635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:21.243567    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 03 20:28:21.243848 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:21.243657    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4" podUID="3d2b0564-e201-4bab-9dec-422ecc7955b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 03 20:28:21.385658 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:21.385010    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:28:21.385658 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:21.385066    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt" podUID="5a14b4da-8ce7-457e-89b8-6e4788d924f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 03 20:28:21.792478 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:21.784248    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 03 20:28:21.792478 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:21.784307    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69" podUID="d6bfef45-f4d7-4482-bde4-2328d5e54b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 03 20:28:21.792839 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:21.661668    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4" event={"ID":"29af63ab-34a3-4ee3-9a74-b8108503642e","Type":"ContainerStarted","Data":"299b38251681a7014b217ca8e00e3aac38bf96e66492f84cd2c6a9ad89ac04ab"}
Nov 03 20:28:21.796131 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:21.796103    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:28:21.817009 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:21.796261    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:28:21.817318 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:21.817272    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6" podUID="ad241f3c-90c0-4f0f-b90c-47eec0af0245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 03 20:28:21.820879 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:21.820581    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4" podUID="29af63ab-34a3-4ee3-9a74-b8108503642e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 20:28:21.918993 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:21.918960    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69" event={"ID":"d6bfef45-f4d7-4482-bde4-2328d5e54b6c","Type":"ContainerStarted","Data":"b62b11e0268b8b598c2358b21336c2c65c6b0b5ad6b08ab13b1e33daf4a7e002"}
Nov 03 20:28:22.279088 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:22.276626    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 03 20:28:22.279088 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:22.276687    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4" podUID="3d2b0564-e201-4bab-9dec-422ecc7955b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 03 20:28:22.332628 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:22.317750    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:28:22.332628 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:22.321665    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt" podUID="5a14b4da-8ce7-457e-89b8-6e4788d924f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 03 20:28:22.589809 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:22.589681    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:22.690337 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:22.690295    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:28:22.690548 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:22.690371    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6" podUID="ad241f3c-90c0-4f0f-b90c-47eec0af0245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 03 20:28:22.729819 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:22.716078    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 03 20:28:22.729819 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:22.716137    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69" podUID="d6bfef45-f4d7-4482-bde4-2328d5e54b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 03 20:28:22.743833 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:22.743797    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:28:22.795637 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:22.791646    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4" podUID="29af63ab-34a3-4ee3-9a74-b8108503642e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 20:28:22.851798 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:22.851689    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:28:22.883624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:22.871902    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 20:28:22.883624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:22.872846    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:23.092201 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:23.092156    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 03 20:28:23.319629 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:23.246675    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 03 20:28:23.437653 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:23.415775    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4" podUID="3d2b0564-e201-4bab-9dec-422ecc7955b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 03 20:28:23.437653 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:23.390371    4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:23.437653 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:23.291072    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:28:23.437653 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:23.416206    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt" podUID="5a14b4da-8ce7-457e-89b8-6e4788d924f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 03 20:28:23.602488 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:23.602388    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:23.603577 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:23.603548    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:23.623682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:23.618623    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:28:23.713654 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:23.713487    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 03 20:28:23.713654 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:23.713550    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69" podUID="d6bfef45-f4d7-4482-bde4-2328d5e54b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 03 20:28:23.717626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:23.714036    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:28:23.717626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:23.714080    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6" podUID="ad241f3c-90c0-4f0f-b90c-47eec0af0245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 03 20:28:23.760760 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:23.760724    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:28:23.761038 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:23.761004    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4" podUID="29af63ab-34a3-4ee3-9a74-b8108503642e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 20:28:23.883686 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:23.865179    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:23.910654 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:23.894892    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:28:23.912731 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:23.883930    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:28:24.050624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:23.878981    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:24.094694 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:24.094649    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:24.291643 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:24.291584    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 03 20:28:24.291951 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:24.291912    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4" podUID="3d2b0564-e201-4bab-9dec-422ecc7955b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 03 20:28:24.401070 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:24.364055    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:28:24.401070 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:24.364111    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt" podUID="5a14b4da-8ce7-457e-89b8-6e4788d924f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 03 20:28:24.624849 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:24.624731    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:24.758640 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:24.745433    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:28:24.758640 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:24.745491    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6" podUID="ad241f3c-90c0-4f0f-b90c-47eec0af0245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 03 20:28:24.758640 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:24.755730    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 03 20:28:24.758640 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:24.755801    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69" podUID="d6bfef45-f4d7-4482-bde4-2328d5e54b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 03 20:28:24.758640 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:24.757744    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:28:24.758640 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:24.757797    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4" podUID="29af63ab-34a3-4ee3-9a74-b8108503642e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 20:28:24.935713 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:24.922957    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded"
Nov 03 20:28:25.043624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:25.010262    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:28:25.043624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:25.010393    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:28:25.043624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:25.010433    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:28:25.043624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:25.010487    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:28:25.043624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:25.010520    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:28:25.043624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:25.038318    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:28:25.043624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:25.042375    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:25.145621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:25.142760    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:28:25.233625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:25.233172    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:25.277623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:25.276772    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 03 20:28:25.277623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:25.276831    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4" podUID="3d2b0564-e201-4bab-9dec-422ecc7955b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 03 20:28:25.314224 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:25.314191    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:28:25.325633 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:25.320665    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt" podUID="5a14b4da-8ce7-457e-89b8-6e4788d924f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 03 20:28:25.506679 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:25.491241    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:25.740623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:25.738651    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:28:25.784677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:25.754950    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 03 20:28:25.784677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:25.755022    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69" podUID="d6bfef45-f4d7-4482-bde4-2328d5e54b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 03 20:28:25.784677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:25.756345    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:28:25.784677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:25.756382    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4" podUID="29af63ab-34a3-4ee3-9a74-b8108503642e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 20:28:25.784677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:25.776753    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:28:25.784677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:25.776829    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6" podUID="ad241f3c-90c0-4f0f-b90c-47eec0af0245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 03 20:28:26.121281 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:26.121168    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:26.234864 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:26.234815    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:26.261625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:26.236477    4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": context deadline exceeded"
Nov 03 20:28:26.341227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:26.341179    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:28:26.341535 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:26.341506    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt" podUID="5a14b4da-8ce7-457e-89b8-6e4788d924f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 03 20:28:26.363809 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:26.289535    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Nov 03 20:28:26.689461 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:26.689426    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:28:26.791664 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:26.791591    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6" podUID="ad241f3c-90c0-4f0f-b90c-47eec0af0245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 03 20:28:26.792523 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:26.724056    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 03 20:28:26.792713 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:26.792685    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69" podUID="d6bfef45-f4d7-4482-bde4-2328d5e54b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 03 20:28:26.792835 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:26.759349    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:28:26.793045 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:26.792927    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4" podUID="29af63ab-34a3-4ee3-9a74-b8108503642e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 20:28:26.794159 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:26.794136    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 20:28:26.794644 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:26.794621    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:28:27.017428 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:27.017343    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:27.017742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:27.017722    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:28:27.018964 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:27.018947    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:27.021096 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:27.021070    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:28:27.191667 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:27.181478    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:27.191667 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:27.182319    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:27.335688 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:27.332747    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv" podUID="cc9f7589-f4e9-4445-95cb-7b0cc446cee4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded"
Nov 03 20:28:27.346811 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:27.346756    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body=
Nov 03 20:28:27.347131 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:27.347105    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt" podUID="5a14b4da-8ce7-457e-89b8-6e4788d924f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused"
Nov 03 20:28:27.397658 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:27.374182    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:27.416275 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:27.416246    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 20:28:27.524107 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:27.524059    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:27.673678 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:27.671482    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:28:27.695638 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:27.695592    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:28:27.695936 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:27.695900    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6" podUID="ad241f3c-90c0-4f0f-b90c-47eec0af0245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 03 20:28:27.725204 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:27.725171    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:28:27.725449 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:27.725414    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4" podUID="29af63ab-34a3-4ee3-9a74-b8108503642e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 20:28:27.727245 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:27.727220    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 03 20:28:27.727408 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:27.727386    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69" podUID="d6bfef45-f4d7-4482-bde4-2328d5e54b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 03 20:28:28.185847 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:28.185803    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:28.253761 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:28.253718    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:28.279666 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:28.266850    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:28:28.391759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:28.391705    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:28.537666 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:28.488896    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:28.552723 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:28.552681    4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:28.565705 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:28.565676    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:28.604623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:28.600915    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:28:28.656688 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:28.648992    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:28.677788 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:28.677738    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:28:28.817945 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:28.817829    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:28:28.818176 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:28.818149    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6" podUID="ad241f3c-90c0-4f0f-b90c-47eec0af0245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 03 20:28:28.818762 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:28.818736    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 03 20:28:28.818941 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:28.818919    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69" podUID="d6bfef45-f4d7-4482-bde4-2328d5e54b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 03 20:28:28.854223 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:28.854134    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:28:28.854421 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:28.854228    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4" podUID="29af63ab-34a3-4ee3-9a74-b8108503642e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 20:28:28.887521 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:28.887476    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:29.130697 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:29.129917    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 20:28:29.131919 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:29.131852    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 20:28:29.134083 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:29.134064    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 20:28:29.135025 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:29.135010    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:28:29.726639 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:29.714381    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 03 20:28:29.726639 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:29.714458    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6" podUID="ad241f3c-90c0-4f0f-b90c-47eec0af0245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 03 20:28:29.734323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:29.734267    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Nov 03 20:28:29.734489 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:29.734334    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69" podUID="d6bfef45-f4d7-4482-bde4-2328d5e54b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Nov 03 20:28:29.734489 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:29.734421    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Nov 03 20:28:29.734489 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:29.734443    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4" podUID="29af63ab-34a3-4ee3-9a74-b8108503642e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Nov 03 20:28:30.693565 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:30.693531    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:30.693986 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:30.693954    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:30.694727 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:30.694704    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 20:28:30.717683 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:30.717518    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:30.718560 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:30.718537    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:30.719354 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:30.719335    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:28:30.727351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:30.727326    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:30.727671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:30.727557    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:30.728365 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:30.728349    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:28:31.952705 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:31.952656    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"]
Nov 03 20:28:32.010663 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.006303    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"]
Nov 03 20:28:32.010663 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.006429    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.056431 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.056388    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.056636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.056441    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3844071e-3d45-455b-91be-3de079eacafa-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.057166 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.057137    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.058384 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.058360    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.058494 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.058407    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq226\" (UniqueName: \"kubernetes.io/projected/3844071e-3d45-455b-91be-3de079eacafa-kube-api-access-nq226\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.058494 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.058434    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.058494 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.058472    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3844071e-3d45-455b-91be-3de079eacafa-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.058624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.058509    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3844071e-3d45-455b-91be-3de079eacafa-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.058624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.058561    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.159611 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.159574    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.159781 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.159686    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq226\" (UniqueName: \"kubernetes.io/projected/3844071e-3d45-455b-91be-3de079eacafa-kube-api-access-nq226\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.159781 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.159716    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.159781 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.159748    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3844071e-3d45-455b-91be-3de079eacafa-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.159962 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.159783    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3844071e-3d45-455b-91be-3de079eacafa-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.159962 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.159830    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.159962 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.159927    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.160136 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.159978    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.160136 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.160027    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3844071e-3d45-455b-91be-3de079eacafa-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.160243 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.160185    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.160243 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.160197    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.160355 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.160260    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.160577 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.160501    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.160577 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.160513    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3844071e-3d45-455b-91be-3de079eacafa-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.162142 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.162119    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.162417 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.162399    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3844071e-3d45-455b-91be-3de079eacafa-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.174005 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.173981    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3844071e-3d45-455b-91be-3de079eacafa-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.174193 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.174172    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq226\" (UniqueName: \"kubernetes.io/projected/3844071e-3d45-455b-91be-3de079eacafa-kube-api-access-nq226\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.345446 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.345060    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:32.719491 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:28:32.719457    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3844071e_3d45_455b_91be_3de079eacafa.slice/crio-5667df3bb0ba35e584738213a561ae10f7c75536977a00e4ec47b01056cee24e WatchSource:0}: Error finding container 5667df3bb0ba35e584738213a561ae10f7c75536977a00e4ec47b01056cee24e: Status 404 returned error can't find the container with id 5667df3bb0ba35e584738213a561ae10f7c75536977a00e4ec47b01056cee24e
Nov 03 20:28:32.720737 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.720707    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"]
Nov 03 20:28:32.723890 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.723836    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:28:32.724013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.723943    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:28:32.724013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.723993    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:28:32.967474 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.967272    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b" event={"ID":"3844071e-3d45-455b-91be-3de079eacafa","Type":"ContainerStarted","Data":"bf94f44daf6fa955fb98218662473ead8be0652518b77894f77d37dff817a3b1"}
Nov 03 20:28:32.968148 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.968116    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b" event={"ID":"3844071e-3d45-455b-91be-3de079eacafa","Type":"ContainerStarted","Data":"5667df3bb0ba35e584738213a561ae10f7c75536977a00e4ec47b01056cee24e"}
Nov 03 20:28:32.998354 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:32.996421    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b" podStartSLOduration=1.996401279 podStartE2EDuration="1.996401279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:32.993204921 +0000 UTC m=+1643.126333377" watchObservedRunningTime="2026-11-03 20:28:32.996401279 +0000 UTC m=+1643.129529744"
Nov 03 20:28:33.357782 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:33.357674    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:33.358154 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:33.358131    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 03 20:28:33.370807 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:33.370767    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b" podUID="3844071e-3d45-455b-91be-3de079eacafa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 03 20:28:34.362626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:34.356361    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 03 20:28:34.362626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:34.356418    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b" podUID="3844071e-3d45-455b-91be-3de079eacafa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 03 20:28:35.363622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:35.355392    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 03 20:28:35.363622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:35.355455    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b" podUID="3844071e-3d45-455b-91be-3de079eacafa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 03 20:28:36.359625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:36.352913    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 03 20:28:36.359625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:36.352972    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b" podUID="3844071e-3d45-455b-91be-3de079eacafa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 03 20:28:37.366667 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:37.365693    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 03 20:28:37.366667 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:37.365767    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b" podUID="3844071e-3d45-455b-91be-3de079eacafa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 03 20:28:38.367621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:38.366735    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 03 20:28:38.367621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:38.366795    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b" podUID="3844071e-3d45-455b-91be-3de079eacafa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 03 20:28:39.349037 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:39.348997    4421 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 03 20:28:39.349323 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:39.349292    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b" podUID="3844071e-3d45-455b-91be-3de079eacafa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 03 20:28:40.672430 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:40.672379    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:40.678745 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:40.678653    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:40.680930 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:40.680875    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:28:42.059127 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:28:42.058293    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:49.631875 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:28:49.631786    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:29:19.631524 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:29:19.631485    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:29:19.853314 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:19.853275    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"]
Nov 03 20:29:19.859950 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:19.859925    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:19.865257 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:19.865230    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-30e9de-openshift-default-dockercfg-j5fsc\""
Nov 03 20:29:19.871762 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:19.870986    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"]
Nov 03 20:29:19.995163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:19.995133    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-data\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:19.995347 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:19.995175    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e627d8-c0cf-4c48-b165-1da76f3b7043-istiod-ca-cert\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:19.995347 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:19.995302    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-credential-socket\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:19.995347 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:19.995344    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-token\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:19.995457 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:19.995364    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-envoy\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:19.995559 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:19.995538    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-podinfo\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:19.995638 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:19.995587    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-workload-certs\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:19.995718 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:19.995702    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qkg4\" (UniqueName: \"kubernetes.io/projected/79e627d8-c0cf-4c48-b165-1da76f3b7043-kube-api-access-8qkg4\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:19.995778 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:19.995768    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-workload-socket\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:20.097152 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:20.097110    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-credential-socket\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:20.097370 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:20.097190    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-token\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:20.097370 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:20.097212    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-envoy\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:20.097370 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:20.097340    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-podinfo\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:20.097583 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:20.097379    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-workload-certs\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:20.097583 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:20.097416    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qkg4\" (UniqueName: \"kubernetes.io/projected/79e627d8-c0cf-4c48-b165-1da76f3b7043-kube-api-access-8qkg4\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:20.097583 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:20.097499    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-workload-socket\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:20.097583 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:20.097510    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-credential-socket\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:20.097936 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:20.097786    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-data\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:20.097936 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:20.097845    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-workload-certs\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:20.097936 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:20.097856    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e627d8-c0cf-4c48-b165-1da76f3b7043-istiod-ca-cert\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:20.097936 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:20.097921    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-workload-socket\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:20.098166 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:20.098113    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-data\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:20.098427 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:20.098405    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e627d8-c0cf-4c48-b165-1da76f3b7043-istiod-ca-cert\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:20.100472 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:20.100444    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-envoy\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:20.100643 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:20.100628    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-podinfo\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:20.112363 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:20.112339    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-token\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:20.112473 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:20.112416    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qkg4\" (UniqueName: \"kubernetes.io/projected/79e627d8-c0cf-4c48-b165-1da76f3b7043-kube-api-access-8qkg4\") pod \"e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") " pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:20.180369 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:20.180340    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:20.472520 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:20.472481    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"]
Nov 03 20:29:20.473368 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:29:20.473336    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e627d8_c0cf_4c48_b165_1da76f3b7043.slice/crio-9e434d9244331266f90eeb59945875bba44c91eb3d3b33c4f124f5f75f568fb6 WatchSource:0}: Error finding container 9e434d9244331266f90eeb59945875bba44c91eb3d3b33c4f124f5f75f568fb6: Status 404 returned error can't find the container with id 9e434d9244331266f90eeb59945875bba44c91eb3d3b33c4f124f5f75f568fb6
Nov 03 20:29:20.476308 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:20.476272    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:29:20.476431 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:20.476340    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:29:20.476431 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:20.476371    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:29:20.879491 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:20.879392    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s" event={"ID":"79e627d8-c0cf-4c48-b165-1da76f3b7043","Type":"ContainerStarted","Data":"f0dc6b29393ed8b1744798a331d9b12bd05fb547a691812b9f55babe91a3dbee"}
Nov 03 20:29:20.879491 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:20.879455    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s" event={"ID":"79e627d8-c0cf-4c48-b165-1da76f3b7043","Type":"ContainerStarted","Data":"9e434d9244331266f90eeb59945875bba44c91eb3d3b33c4f124f5f75f568fb6"}
Nov 03 20:29:20.910955 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:20.910844    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s" podStartSLOduration=1.910822252 podStartE2EDuration="1.910822252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:20.906876641 +0000 UTC m=+1691.040005104" watchObservedRunningTime="2026-11-03 20:29:20.910822252 +0000 UTC m=+1691.043950715"
Nov 03 20:29:21.181435 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:21.181345    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:21.183318 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:21.183290    4421 patch_prober.go:29] interesting pod/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 03 20:29:21.183434 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:21.183353    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s" podUID="79e627d8-c0cf-4c48-b165-1da76f3b7043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 03 20:29:22.181127 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:22.181096    4421 patch_prober.go:29] interesting pod/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 03 20:29:22.181511 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:22.181148    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s" podUID="79e627d8-c0cf-4c48-b165-1da76f3b7043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 03 20:29:23.181624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:23.181576    4421 patch_prober.go:29] interesting pod/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 03 20:29:23.182072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:23.181712    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s" podUID="79e627d8-c0cf-4c48-b165-1da76f3b7043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 03 20:29:24.180711 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:24.180676    4421 patch_prober.go:29] interesting pod/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 03 20:29:24.180889 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:24.180729    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s" podUID="79e627d8-c0cf-4c48-b165-1da76f3b7043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 03 20:29:25.181507 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:25.181477    4421 patch_prober.go:29] interesting pod/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 03 20:29:25.181916 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:25.181527    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s" podUID="79e627d8-c0cf-4c48-b165-1da76f3b7043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 03 20:29:26.183017 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:26.182979    4421 patch_prober.go:29] interesting pod/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 03 20:29:26.183719 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:26.183591    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s" podUID="79e627d8-c0cf-4c48-b165-1da76f3b7043" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 03 20:29:27.349307 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:27.349260    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:27.349907 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:27.349884    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:27.351864 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:29:27.351840    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:29:49.632105 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:29:49.632068    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:30:00.267137 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:00.267104    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv"]
Nov 03 20:30:00.273810 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:00.273785    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q"]
Nov 03 20:30:00.273978 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:00.273920    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv"
Nov 03 20:30:00.279944 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:00.279919    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv"]
Nov 03 20:30:00.280216 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:00.280193    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q"
Nov 03 20:30:00.280443 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:00.280422    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q"]
Nov 03 20:30:00.288287 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:00.288266    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\""
Nov 03 20:30:00.372273 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:00.372232    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1901a6-6aeb-4786-88a7-1b7836495531-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q\" (UID: \"2f1901a6-6aeb-4786-88a7-1b7836495531\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q"
Nov 03 20:30:00.372489 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:00.372462    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qnpl\" (UniqueName: \"kubernetes.io/projected/2f1901a6-6aeb-4786-88a7-1b7836495531-kube-api-access-9qnpl\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q\" (UID: \"2f1901a6-6aeb-4786-88a7-1b7836495531\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q"
Nov 03 20:30:00.372573 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:00.372523    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f785b3f-e95b-4a67-b7f8-4e1731a7e070-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv\" (UID: \"8f785b3f-e95b-4a67-b7f8-4e1731a7e070\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv"
Nov 03 20:30:00.372741 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:00.372726    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8h9r\" (UniqueName: \"kubernetes.io/projected/8f785b3f-e95b-4a67-b7f8-4e1731a7e070-kube-api-access-b8h9r\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv\" (UID: \"8f785b3f-e95b-4a67-b7f8-4e1731a7e070\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv"
Nov 03 20:30:00.474831 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:00.474800    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qnpl\" (UniqueName: \"kubernetes.io/projected/2f1901a6-6aeb-4786-88a7-1b7836495531-kube-api-access-9qnpl\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q\" (UID: \"2f1901a6-6aeb-4786-88a7-1b7836495531\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q"
Nov 03 20:30:00.475025 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:00.474847    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f785b3f-e95b-4a67-b7f8-4e1731a7e070-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv\" (UID: \"8f785b3f-e95b-4a67-b7f8-4e1731a7e070\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv"
Nov 03 20:30:00.475025 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:00.474945    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8h9r\" (UniqueName: \"kubernetes.io/projected/8f785b3f-e95b-4a67-b7f8-4e1731a7e070-kube-api-access-b8h9r\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv\" (UID: \"8f785b3f-e95b-4a67-b7f8-4e1731a7e070\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv"
Nov 03 20:30:00.475165 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:00.475091    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1901a6-6aeb-4786-88a7-1b7836495531-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q\" (UID: \"2f1901a6-6aeb-4786-88a7-1b7836495531\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q"
Nov 03 20:30:00.475636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:00.475572    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f785b3f-e95b-4a67-b7f8-4e1731a7e070-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv\" (UID: \"8f785b3f-e95b-4a67-b7f8-4e1731a7e070\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv"
Nov 03 20:30:00.475774 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:00.475751    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1901a6-6aeb-4786-88a7-1b7836495531-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q\" (UID: \"2f1901a6-6aeb-4786-88a7-1b7836495531\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q"
Nov 03 20:30:00.488815 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:00.488790    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qnpl\" (UniqueName: \"kubernetes.io/projected/2f1901a6-6aeb-4786-88a7-1b7836495531-kube-api-access-9qnpl\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q\" (UID: \"2f1901a6-6aeb-4786-88a7-1b7836495531\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q"
Nov 03 20:30:00.488982 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:00.488961    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8h9r\" (UniqueName: \"kubernetes.io/projected/8f785b3f-e95b-4a67-b7f8-4e1731a7e070-kube-api-access-b8h9r\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv\" (UID: \"8f785b3f-e95b-4a67-b7f8-4e1731a7e070\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv"
Nov 03 20:30:00.598672 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:00.598590    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv"
Nov 03 20:30:00.627467 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:00.627429    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q"
Nov 03 20:30:01.085671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:01.085635    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv"]
Nov 03 20:30:01.086035 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:30:01.086009    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f785b3f_e95b_4a67_b7f8_4e1731a7e070.slice/crio-7ed403207f427798645a973baa34ed7839a4e1d9a20bcb382b64fa1274f82893 WatchSource:0}: Error finding container 7ed403207f427798645a973baa34ed7839a4e1d9a20bcb382b64fa1274f82893: Status 404 returned error can't find the container with id 7ed403207f427798645a973baa34ed7839a4e1d9a20bcb382b64fa1274f82893
Nov 03 20:30:01.086957 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:30:01.086938    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1901a6_6aeb_4786_88a7_1b7836495531.slice/crio-f505da5ca24754f20a5790fd5db3e755eb9d9464fc0a92eff24abb526c707fb2 WatchSource:0}: Error finding container f505da5ca24754f20a5790fd5db3e755eb9d9464fc0a92eff24abb526c707fb2: Status 404 returned error can't find the container with id f505da5ca24754f20a5790fd5db3e755eb9d9464fc0a92eff24abb526c707fb2
Nov 03 20:30:01.093760 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:01.092473    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q"]
Nov 03 20:30:01.408005 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:01.407702    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv" event={"ID":"8f785b3f-e95b-4a67-b7f8-4e1731a7e070","Type":"ContainerStarted","Data":"7ed403207f427798645a973baa34ed7839a4e1d9a20bcb382b64fa1274f82893"}
Nov 03 20:30:01.410755 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:01.410725    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q" event={"ID":"2f1901a6-6aeb-4786-88a7-1b7836495531","Type":"ContainerStarted","Data":"f505da5ca24754f20a5790fd5db3e755eb9d9464fc0a92eff24abb526c707fb2"}
Nov 03 20:30:02.421030 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:02.420943    4421 generic.go:356] "Generic (PLEG): container finished" podID="8f785b3f-e95b-4a67-b7f8-4e1731a7e070" containerID="f0955c75e3d0ad1583b9c9fbbe05037d548f179639fbf5c8d1ea226800a816b8" exitCode=60
Nov 03 20:30:02.421481 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:02.421036    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv" event={"ID":"8f785b3f-e95b-4a67-b7f8-4e1731a7e070","Type":"ContainerDied","Data":"f0955c75e3d0ad1583b9c9fbbe05037d548f179639fbf5c8d1ea226800a816b8"}
Nov 03 20:30:02.421481 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:02.421358    4421 scope.go:122] "RemoveContainer" containerID="f0955c75e3d0ad1583b9c9fbbe05037d548f179639fbf5c8d1ea226800a816b8"
Nov 03 20:30:02.422951 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:02.422928    4421 generic.go:356] "Generic (PLEG): container finished" podID="2f1901a6-6aeb-4786-88a7-1b7836495531" containerID="6abbd4c8938dfe931b0e670803702a150e918574c6062ec3d44d4d3460f68f45" exitCode=60
Nov 03 20:30:02.423053 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:02.423002    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q" event={"ID":"2f1901a6-6aeb-4786-88a7-1b7836495531","Type":"ContainerDied","Data":"6abbd4c8938dfe931b0e670803702a150e918574c6062ec3d44d4d3460f68f45"}
Nov 03 20:30:02.423301 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:02.423285    4421 scope.go:122] "RemoveContainer" containerID="6abbd4c8938dfe931b0e670803702a150e918574c6062ec3d44d4d3460f68f45"
Nov 03 20:30:03.450082 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:03.450046    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv" event={"ID":"8f785b3f-e95b-4a67-b7f8-4e1731a7e070","Type":"ContainerStarted","Data":"900c8a42dfebad67b3838d5e4b54dda973605ad8473e7c53974454b2e2d4d9df"}
Nov 03 20:30:03.452944 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:03.452917    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q" event={"ID":"2f1901a6-6aeb-4786-88a7-1b7836495531","Type":"ContainerStarted","Data":"3ba2a74729dfec13c4d39a56bf2fc8c905f90ad062ba7f0576da4b2b01d0398d"}
Nov 03 20:30:03.472577 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:03.472531    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv" podStartSLOduration=3.472518108 podStartE2EDuration="3.472518108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:03.468646241 +0000 UTC m=+1733.601774703" watchObservedRunningTime="2026-11-03 20:30:03.472518108 +0000 UTC m=+1733.605646569"
Nov 03 20:30:03.491897 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:03.491856    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q" podStartSLOduration=3.491843583 podStartE2EDuration="3.491843583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:03.487451146 +0000 UTC m=+1733.620579608" watchObservedRunningTime="2026-11-03 20:30:03.491843583 +0000 UTC m=+1733.624972044"
Nov 03 20:30:04.463971 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:04.463930    4421 generic.go:356] "Generic (PLEG): container finished" podID="8f785b3f-e95b-4a67-b7f8-4e1731a7e070" containerID="900c8a42dfebad67b3838d5e4b54dda973605ad8473e7c53974454b2e2d4d9df" exitCode=60
Nov 03 20:30:04.464473 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:04.464004    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv" event={"ID":"8f785b3f-e95b-4a67-b7f8-4e1731a7e070","Type":"ContainerDied","Data":"900c8a42dfebad67b3838d5e4b54dda973605ad8473e7c53974454b2e2d4d9df"}
Nov 03 20:30:04.464473 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:04.464075    4421 scope.go:122] "RemoveContainer" containerID="f0955c75e3d0ad1583b9c9fbbe05037d548f179639fbf5c8d1ea226800a816b8"
Nov 03 20:30:04.464473 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:04.464420    4421 scope.go:122] "RemoveContainer" containerID="900c8a42dfebad67b3838d5e4b54dda973605ad8473e7c53974454b2e2d4d9df"
Nov 03 20:30:04.464905 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:30:04.464882    4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv_odh-ai-gateway-infra(8f785b3f-e95b-4a67-b7f8-4e1731a7e070)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv" podUID="8f785b3f-e95b-4a67-b7f8-4e1731a7e070"
Nov 03 20:30:04.466383 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:04.466362    4421 generic.go:356] "Generic (PLEG): container finished" podID="2f1901a6-6aeb-4786-88a7-1b7836495531" containerID="3ba2a74729dfec13c4d39a56bf2fc8c905f90ad062ba7f0576da4b2b01d0398d" exitCode=60
Nov 03 20:30:04.466557 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:04.466442    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q" event={"ID":"2f1901a6-6aeb-4786-88a7-1b7836495531","Type":"ContainerDied","Data":"3ba2a74729dfec13c4d39a56bf2fc8c905f90ad062ba7f0576da4b2b01d0398d"}
Nov 03 20:30:04.466800 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:04.466784    4421 scope.go:122] "RemoveContainer" containerID="3ba2a74729dfec13c4d39a56bf2fc8c905f90ad062ba7f0576da4b2b01d0398d"
Nov 03 20:30:04.467205 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:30:04.467186    4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q_odh-ai-gateway-infra(2f1901a6-6aeb-4786-88a7-1b7836495531)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q" podUID="2f1901a6-6aeb-4786-88a7-1b7836495531"
Nov 03 20:30:04.478369 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:04.478349    4421 scope.go:122] "RemoveContainer" containerID="6abbd4c8938dfe931b0e670803702a150e918574c6062ec3d44d4d3460f68f45"
Nov 03 20:30:05.477313 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:05.477285    4421 scope.go:122] "RemoveContainer" containerID="900c8a42dfebad67b3838d5e4b54dda973605ad8473e7c53974454b2e2d4d9df"
Nov 03 20:30:05.477801 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:30:05.477745    4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv_odh-ai-gateway-infra(8f785b3f-e95b-4a67-b7f8-4e1731a7e070)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv" podUID="8f785b3f-e95b-4a67-b7f8-4e1731a7e070"
Nov 03 20:30:05.478993 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:05.478976    4421 scope.go:122] "RemoveContainer" containerID="3ba2a74729dfec13c4d39a56bf2fc8c905f90ad062ba7f0576da4b2b01d0398d"
Nov 03 20:30:05.479298 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:30:05.479278    4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q_odh-ai-gateway-infra(2f1901a6-6aeb-4786-88a7-1b7836495531)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q" podUID="2f1901a6-6aeb-4786-88a7-1b7836495531"
Nov 03 20:30:11.961392 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:11.961358    4421 scope.go:122] "RemoveContainer" containerID="23b5c1007d4b26ed38f71376dc5e8e21b66eaca3256c8a1b326ae3ba4431b74c"
Nov 03 20:30:15.009579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:15.009547    4421 scope.go:122] "RemoveContainer" containerID="3ba2a74729dfec13c4d39a56bf2fc8c905f90ad062ba7f0576da4b2b01d0398d"
Nov 03 20:30:15.010101 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:15.009760    4421 scope.go:122] "RemoveContainer" containerID="900c8a42dfebad67b3838d5e4b54dda973605ad8473e7c53974454b2e2d4d9df"
Nov 03 20:30:15.561711 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:15.561630    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv" event={"ID":"8f785b3f-e95b-4a67-b7f8-4e1731a7e070","Type":"ContainerStarted","Data":"8435aa75b333774a5a779e96c90bb397aed94c2f53bc35a04a0abe8860723dcf"}
Nov 03 20:30:15.563751 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:15.563724    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q" event={"ID":"2f1901a6-6aeb-4786-88a7-1b7836495531","Type":"ContainerStarted","Data":"15a0bc706db76bd8f18a561ff9bc5ed5e4b876eda570a47878fb0ad46b9feeab"}
Nov 03 20:30:16.632586 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:16.632047    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv"]
Nov 03 20:30:16.633739 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:16.633712    4421 generic.go:356] "Generic (PLEG): container finished" podID="8f785b3f-e95b-4a67-b7f8-4e1731a7e070" containerID="8435aa75b333774a5a779e96c90bb397aed94c2f53bc35a04a0abe8860723dcf" exitCode=60
Nov 03 20:30:16.633907 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:16.633820    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv" event={"ID":"8f785b3f-e95b-4a67-b7f8-4e1731a7e070","Type":"ContainerDied","Data":"8435aa75b333774a5a779e96c90bb397aed94c2f53bc35a04a0abe8860723dcf"}
Nov 03 20:30:16.634002 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:16.633928    4421 scope.go:122] "RemoveContainer" containerID="900c8a42dfebad67b3838d5e4b54dda973605ad8473e7c53974454b2e2d4d9df"
Nov 03 20:30:16.638826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:16.638429    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q"]
Nov 03 20:30:16.641779 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:16.641753    4421 generic.go:356] "Generic (PLEG): container finished" podID="2f1901a6-6aeb-4786-88a7-1b7836495531" containerID="15a0bc706db76bd8f18a561ff9bc5ed5e4b876eda570a47878fb0ad46b9feeab" exitCode=60
Nov 03 20:30:16.642013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:16.642001    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q" event={"ID":"2f1901a6-6aeb-4786-88a7-1b7836495531","Type":"ContainerDied","Data":"15a0bc706db76bd8f18a561ff9bc5ed5e4b876eda570a47878fb0ad46b9feeab"}
Nov 03 20:30:16.655394 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:16.655367    4421 scope.go:122] "RemoveContainer" containerID="3ba2a74729dfec13c4d39a56bf2fc8c905f90ad062ba7f0576da4b2b01d0398d"
Nov 03 20:30:16.859188 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:16.859159    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv"
Nov 03 20:30:16.885779 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:16.885704    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q"
Nov 03 20:30:16.891936 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:16.891911    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f785b3f-e95b-4a67-b7f8-4e1731a7e070-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f785b3f-e95b-4a67-b7f8-4e1731a7e070-maas-api-service-ca\") pod \"8f785b3f-e95b-4a67-b7f8-4e1731a7e070\" (UID: \"8f785b3f-e95b-4a67-b7f8-4e1731a7e070\") "
Nov 03 20:30:16.892084 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:16.892007    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f785b3f-e95b-4a67-b7f8-4e1731a7e070-kube-api-access-b8h9r\" (UniqueName: \"kubernetes.io/projected/8f785b3f-e95b-4a67-b7f8-4e1731a7e070-kube-api-access-b8h9r\") pod \"8f785b3f-e95b-4a67-b7f8-4e1731a7e070\" (UID: \"8f785b3f-e95b-4a67-b7f8-4e1731a7e070\") "
Nov 03 20:30:16.892273 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:16.892249    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f785b3f-e95b-4a67-b7f8-4e1731a7e070-maas-api-service-ca" pod "8f785b3f-e95b-4a67-b7f8-4e1731a7e070" (UID: "8f785b3f-e95b-4a67-b7f8-4e1731a7e070"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:16.893198 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:16.893175    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f785b3f-e95b-4a67-b7f8-4e1731a7e070-maas-api-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:16.895874 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:16.895848    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f785b3f-e95b-4a67-b7f8-4e1731a7e070-kube-api-access-b8h9r" pod "8f785b3f-e95b-4a67-b7f8-4e1731a7e070" (UID: "8f785b3f-e95b-4a67-b7f8-4e1731a7e070"). InnerVolumeSpecName "kube-api-access-b8h9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:16.994744 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:16.994711    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1901a6-6aeb-4786-88a7-1b7836495531-kube-api-access-9qnpl\" (UniqueName: \"kubernetes.io/projected/2f1901a6-6aeb-4786-88a7-1b7836495531-kube-api-access-9qnpl\") pod \"2f1901a6-6aeb-4786-88a7-1b7836495531\" (UID: \"2f1901a6-6aeb-4786-88a7-1b7836495531\") "
Nov 03 20:30:16.994940 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:16.994922    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f1901a6-6aeb-4786-88a7-1b7836495531-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1901a6-6aeb-4786-88a7-1b7836495531-maas-api-service-ca\") pod \"2f1901a6-6aeb-4786-88a7-1b7836495531\" (UID: \"2f1901a6-6aeb-4786-88a7-1b7836495531\") "
Nov 03 20:30:16.995242 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:16.995218    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1901a6-6aeb-4786-88a7-1b7836495531-maas-api-service-ca" pod "2f1901a6-6aeb-4786-88a7-1b7836495531" (UID: "2f1901a6-6aeb-4786-88a7-1b7836495531"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:30:16.995772 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:16.995751    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8h9r\" (UniqueName: \"kubernetes.io/projected/8f785b3f-e95b-4a67-b7f8-4e1731a7e070-kube-api-access-b8h9r\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:16.995857 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:16.995775    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1901a6-6aeb-4786-88a7-1b7836495531-maas-api-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:16.996978 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:16.996959    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1901a6-6aeb-4786-88a7-1b7836495531-kube-api-access-9qnpl" pod "2f1901a6-6aeb-4786-88a7-1b7836495531" (UID: "2f1901a6-6aeb-4786-88a7-1b7836495531"). InnerVolumeSpecName "kube-api-access-9qnpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:17.098239 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:17.098211    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qnpl\" (UniqueName: \"kubernetes.io/projected/2f1901a6-6aeb-4786-88a7-1b7836495531-kube-api-access-9qnpl\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:17.651805 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:17.651773    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv"
Nov 03 20:30:17.651805 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:17.651792    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv" event={"ID":"8f785b3f-e95b-4a67-b7f8-4e1731a7e070","Type":"ContainerDied","Data":"7ed403207f427798645a973baa34ed7839a4e1d9a20bcb382b64fa1274f82893"}
Nov 03 20:30:17.652373 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:17.651856    4421 scope.go:122] "RemoveContainer" containerID="8435aa75b333774a5a779e96c90bb397aed94c2f53bc35a04a0abe8860723dcf"
Nov 03 20:30:17.655180 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:17.655164    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q"
Nov 03 20:30:17.655341 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:17.655203    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q" event={"ID":"2f1901a6-6aeb-4786-88a7-1b7836495531","Type":"ContainerDied","Data":"f505da5ca24754f20a5790fd5db3e755eb9d9464fc0a92eff24abb526c707fb2"}
Nov 03 20:30:17.666459 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:17.666421    4421 scope.go:122] "RemoveContainer" containerID="15a0bc706db76bd8f18a561ff9bc5ed5e4b876eda570a47878fb0ad46b9feeab"
Nov 03 20:30:17.683422 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:17.683397    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q"]
Nov 03 20:30:17.688085 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:17.688064    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3c25bf-29895630-l845q"]
Nov 03 20:30:17.700191 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:17.700164    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv"]
Nov 03 20:30:17.704494 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:17.704471    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895630-bkjmv"]
Nov 03 20:30:18.018165 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:18.018130    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1901a6-6aeb-4786-88a7-1b7836495531" path="/var/lib/kubelet/pods/2f1901a6-6aeb-4786-88a7-1b7836495531/volumes"
Nov 03 20:30:18.018660 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:18.018644    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f785b3f-e95b-4a67-b7f8-4e1731a7e070" path="/var/lib/kubelet/pods/8f785b3f-e95b-4a67-b7f8-4e1731a7e070/volumes"
Nov 03 20:30:19.726195 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:30:19.726058    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:30:29.045625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:29.044882    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:29.045625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:29.045334    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Nov 03 20:30:29.143657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:29.142168    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 03 20:30:30.075624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:30.075499    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:30.108438 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:30.107064    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:30:30.119690 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:30.119654    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 20:30:30.166659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:30.152936    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:30.166659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:30.153105    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:30:30.294914 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:30.291911    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 20:30:48.466836 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.466784    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"]
Nov 03 20:30:48.468033 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.468012    4421 cpu_manager.go:414] "containerMap: removing container" podUID="8f785b3f-e95b-4a67-b7f8-4e1731a7e070" containerName="cleanup"
Nov 03 20:30:48.468163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.468034    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f785b3f-e95b-4a67-b7f8-4e1731a7e070" containerName="cleanup"
Nov 03 20:30:48.468163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.468064    4421 cpu_manager.go:414] "containerMap: removing container" podUID="2f1901a6-6aeb-4786-88a7-1b7836495531" containerName="cleanup"
Nov 03 20:30:48.468163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.468074    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1901a6-6aeb-4786-88a7-1b7836495531" containerName="cleanup"
Nov 03 20:30:48.468163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.468106    4421 cpu_manager.go:414] "containerMap: removing container" podUID="8f785b3f-e95b-4a67-b7f8-4e1731a7e070" containerName="cleanup"
Nov 03 20:30:48.468163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.468114    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f785b3f-e95b-4a67-b7f8-4e1731a7e070" containerName="cleanup"
Nov 03 20:30:48.468163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.468130    4421 cpu_manager.go:414] "containerMap: removing container" podUID="8f785b3f-e95b-4a67-b7f8-4e1731a7e070" containerName="cleanup"
Nov 03 20:30:48.468163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.468136    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f785b3f-e95b-4a67-b7f8-4e1731a7e070" containerName="cleanup"
Nov 03 20:30:48.468163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.468150    4421 cpu_manager.go:414] "containerMap: removing container" podUID="2f1901a6-6aeb-4786-88a7-1b7836495531" containerName="cleanup"
Nov 03 20:30:48.468163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.468157    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1901a6-6aeb-4786-88a7-1b7836495531" containerName="cleanup"
Nov 03 20:30:48.468735 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.468377    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1901a6-6aeb-4786-88a7-1b7836495531" containerName="cleanup"
Nov 03 20:30:48.468735 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.468395    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f785b3f-e95b-4a67-b7f8-4e1731a7e070" containerName="cleanup"
Nov 03 20:30:48.468735 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.468415    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f785b3f-e95b-4a67-b7f8-4e1731a7e070" containerName="cleanup"
Nov 03 20:30:48.468735 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.468423    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1901a6-6aeb-4786-88a7-1b7836495531" containerName="cleanup"
Nov 03 20:30:48.473189 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.473164    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.492529 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.492500    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"]
Nov 03 20:30:48.553453 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.553417    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.553676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.553482    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.553676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.553590    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.553841 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.553697    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.553841 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.553725    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.553841 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.553820    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.554011 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.553948    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.554114 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.554093    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.554201 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.554146    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbk9l\" (UniqueName: \"kubernetes.io/projected/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-kube-api-access-gbk9l\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.655721 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.655688    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.655721 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.655724    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.656003 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.655789    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.656003 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.655901    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.656127 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.656034    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.656127 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.656078    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbk9l\" (UniqueName: \"kubernetes.io/projected/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-kube-api-access-gbk9l\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.656223 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.656185    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.656266 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.656237    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.656335 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.656318    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.656802 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.656775    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.657108 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.657061    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.657214 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.657113    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.657439 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.657414    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.657692 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.657666    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.658325 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.658303    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.658756 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.658740    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.670943 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.670919    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.672501 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.672481    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbk9l\" (UniqueName: \"kubernetes.io/projected/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-kube-api-access-gbk9l\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:48.797135 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:48.797044    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:49.190910 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:30:49.190876    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d17b9b_ecc0_471b_ae3e_30b76b7241ce.slice/crio-bc505efacb314a2351c428a09aec68618627948bf8a5aaf0cc362c3310e770e9 WatchSource:0}: Error finding container bc505efacb314a2351c428a09aec68618627948bf8a5aaf0cc362c3310e770e9: Status 404 returned error can't find the container with id bc505efacb314a2351c428a09aec68618627948bf8a5aaf0cc362c3310e770e9
Nov 03 20:30:49.191473 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:49.191372    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"]
Nov 03 20:30:49.194436 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:49.194397    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:30:49.194524 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:49.194485    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:30:49.194566 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:49.194534    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:30:49.211359 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:49.211330    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw" event={"ID":"47d17b9b-ecc0-471b-ae3e-30b76b7241ce","Type":"ContainerStarted","Data":"bc505efacb314a2351c428a09aec68618627948bf8a5aaf0cc362c3310e770e9"}
Nov 03 20:30:49.632561 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:30:49.632479    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:30:50.223691 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:50.223641    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw" event={"ID":"47d17b9b-ecc0-471b-ae3e-30b76b7241ce","Type":"ContainerStarted","Data":"c9ea841ef292767c409a7cd168fb4615091160264bf906dee439a0747dc0d842"}
Nov 03 20:30:50.273919 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:50.273860    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw" podStartSLOduration=2.2738447600000002 podStartE2EDuration="2.27384476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:50.265396685 +0000 UTC m=+1780.398525151" watchObservedRunningTime="2026-11-03 20:30:50.27384476 +0000 UTC m=+1780.406973222"
Nov 03 20:30:50.800095 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:50.799828    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:50.801023 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:50.800993    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Nov 03 20:30:50.801231 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:50.801202    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw" podUID="47d17b9b-ecc0-471b-ae3e-30b76b7241ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Nov 03 20:30:51.802625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:51.802159    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Nov 03 20:30:51.802625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:51.802245    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw" podUID="47d17b9b-ecc0-471b-ae3e-30b76b7241ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Nov 03 20:30:52.797879 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:52.797845    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Nov 03 20:30:52.798068 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:52.797921    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw" podUID="47d17b9b-ecc0-471b-ae3e-30b76b7241ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Nov 03 20:30:53.797637 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:53.797607    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Nov 03 20:30:53.798037 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:53.797682    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw" podUID="47d17b9b-ecc0-471b-ae3e-30b76b7241ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Nov 03 20:30:54.797807 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:54.797752    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Nov 03 20:30:54.798213 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:54.797843    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw" podUID="47d17b9b-ecc0-471b-ae3e-30b76b7241ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Nov 03 20:30:55.859047 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:55.856968    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:55.859047 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:55.858888    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:30:55.867823 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:30:55.867783    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:31:06.051535 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:06.051500    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"]
Nov 03 20:31:06.052270 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:06.052251    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"]
Nov 03 20:31:06.052871 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:06.052640    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47" podUID="b270c5c6-0444-4410-a70d-37edb799adce" containerName="istio-proxy" containerID="cri-o://8cd02ec096fa460a4056a45596bddfb301f6dd5867c22a355a4c8d44d4c0052d" gracePeriod=30
Nov 03 20:31:06.617626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:06.612322    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx" podUID="484992e4-0e7a-4728-9adb-0106b20d4dc0" containerName="istio-proxy" containerID="cri-o://156e81e44712c985e6dde2291cecf41fc8ca94fe9917ea0df651460cfac77979" gracePeriod=30
Nov 03 20:31:11.367280 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.367252    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:31:11.482368 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.482335    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b270c5c6-0444-4410-a70d-37edb799adce-kube-api-access-xs6l9\" (UniqueName: \"kubernetes.io/projected/b270c5c6-0444-4410-a70d-37edb799adce-kube-api-access-xs6l9\") pod \"b270c5c6-0444-4410-a70d-37edb799adce\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") "
Nov 03 20:31:11.482665 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.482433    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b270c5c6-0444-4410-a70d-37edb799adce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b270c5c6-0444-4410-a70d-37edb799adce-istio-podinfo\") pod \"b270c5c6-0444-4410-a70d-37edb799adce\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") "
Nov 03 20:31:11.482665 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.482565    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-workload-certs\") pod \"b270c5c6-0444-4410-a70d-37edb799adce\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") "
Nov 03 20:31:11.482830 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.482686    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-istio-envoy\") pod \"b270c5c6-0444-4410-a70d-37edb799adce\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") "
Nov 03 20:31:11.482830 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.482723    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b270c5c6-0444-4410-a70d-37edb799adce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b270c5c6-0444-4410-a70d-37edb799adce-istiod-ca-cert\") pod \"b270c5c6-0444-4410-a70d-37edb799adce\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") "
Nov 03 20:31:11.482956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.482839    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b270c5c6-0444-4410-a70d-37edb799adce-istio-token\" (UniqueName: \"kubernetes.io/projected/b270c5c6-0444-4410-a70d-37edb799adce-istio-token\") pod \"b270c5c6-0444-4410-a70d-37edb799adce\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") "
Nov 03 20:31:11.482956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.482945    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-istio-data\") pod \"b270c5c6-0444-4410-a70d-37edb799adce\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") "
Nov 03 20:31:11.483067 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.482983    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-credential-socket\") pod \"b270c5c6-0444-4410-a70d-37edb799adce\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") "
Nov 03 20:31:11.483067 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.483033    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-workload-socket\") pod \"b270c5c6-0444-4410-a70d-37edb799adce\" (UID: \"b270c5c6-0444-4410-a70d-37edb799adce\") "
Nov 03 20:31:11.483067 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.483038    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-workload-certs" pod "b270c5c6-0444-4410-a70d-37edb799adce" (UID: "b270c5c6-0444-4410-a70d-37edb799adce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:11.483240 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.483124    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b270c5c6-0444-4410-a70d-37edb799adce-istiod-ca-cert" pod "b270c5c6-0444-4410-a70d-37edb799adce" (UID: "b270c5c6-0444-4410-a70d-37edb799adce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:11.483297 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.483258    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-istio-data" pod "b270c5c6-0444-4410-a70d-37edb799adce" (UID: "b270c5c6-0444-4410-a70d-37edb799adce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:11.483355 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.483265    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-credential-socket" pod "b270c5c6-0444-4410-a70d-37edb799adce" (UID: "b270c5c6-0444-4410-a70d-37edb799adce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:11.483458 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.483425    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-workload-socket" pod "b270c5c6-0444-4410-a70d-37edb799adce" (UID: "b270c5c6-0444-4410-a70d-37edb799adce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:11.484120 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.484094    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:11.484212 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.484127    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b270c5c6-0444-4410-a70d-37edb799adce-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:11.484212 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.484143    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:11.484212 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.484157    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:11.484212 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.484172    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:11.485154 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.485125    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b270c5c6-0444-4410-a70d-37edb799adce-istio-podinfo" pod "b270c5c6-0444-4410-a70d-37edb799adce" (UID: "b270c5c6-0444-4410-a70d-37edb799adce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:31:11.485249 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.485192    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b270c5c6-0444-4410-a70d-37edb799adce-istio-token" pod "b270c5c6-0444-4410-a70d-37edb799adce" (UID: "b270c5c6-0444-4410-a70d-37edb799adce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:11.485293 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.485251    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-istio-envoy" pod "b270c5c6-0444-4410-a70d-37edb799adce" (UID: "b270c5c6-0444-4410-a70d-37edb799adce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:11.485336 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.485321    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b270c5c6-0444-4410-a70d-37edb799adce-kube-api-access-xs6l9" pod "b270c5c6-0444-4410-a70d-37edb799adce" (UID: "b270c5c6-0444-4410-a70d-37edb799adce"). InnerVolumeSpecName "kube-api-access-xs6l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:11.617900 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.617799    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs6l9\" (UniqueName: \"kubernetes.io/projected/b270c5c6-0444-4410-a70d-37edb799adce-kube-api-access-xs6l9\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:11.617900 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.617838    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b270c5c6-0444-4410-a70d-37edb799adce-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:11.617900 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.617853    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b270c5c6-0444-4410-a70d-37edb799adce-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:11.617900 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.617866    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b270c5c6-0444-4410-a70d-37edb799adce-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:11.662824 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.662789    4421 generic.go:356] "Generic (PLEG): container finished" podID="b270c5c6-0444-4410-a70d-37edb799adce" containerID="8cd02ec096fa460a4056a45596bddfb301f6dd5867c22a355a4c8d44d4c0052d" exitCode=0
Nov 03 20:31:11.663007 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.662853    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"
Nov 03 20:31:11.663007 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.662873    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47" event={"ID":"b270c5c6-0444-4410-a70d-37edb799adce","Type":"ContainerDied","Data":"8cd02ec096fa460a4056a45596bddfb301f6dd5867c22a355a4c8d44d4c0052d"}
Nov 03 20:31:11.663007 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.662913    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47" event={"ID":"b270c5c6-0444-4410-a70d-37edb799adce","Type":"ContainerDied","Data":"0c9722064644a7ef9ec7c09a4599d0391704ad7a921790ac7a0ff6963ada8f61"}
Nov 03 20:31:11.663007 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.662939    4421 scope.go:122] "RemoveContainer" containerID="8cd02ec096fa460a4056a45596bddfb301f6dd5867c22a355a4c8d44d4c0052d"
Nov 03 20:31:11.702843 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.702818    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"]
Nov 03 20:31:11.707136 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.707114    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-ccq47"]
Nov 03 20:31:11.733049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.733029    4421 scope.go:122] "RemoveContainer" containerID="2884e4fab020ea09b5e6f5d307306ff6e6778c87b614512d886f9c67b009a517"
Nov 03 20:31:11.746088 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.746063    4421 scope.go:122] "RemoveContainer" containerID="8cd02ec096fa460a4056a45596bddfb301f6dd5867c22a355a4c8d44d4c0052d"
Nov 03 20:31:11.746436 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:31:11.746410    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cd02ec096fa460a4056a45596bddfb301f6dd5867c22a355a4c8d44d4c0052d\": container with ID starting with 8cd02ec096fa460a4056a45596bddfb301f6dd5867c22a355a4c8d44d4c0052d not found: ID does not exist" containerID="8cd02ec096fa460a4056a45596bddfb301f6dd5867c22a355a4c8d44d4c0052d"
Nov 03 20:31:11.746500 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.746449    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cd02ec096fa460a4056a45596bddfb301f6dd5867c22a355a4c8d44d4c0052d"} err="failed to get container status \"8cd02ec096fa460a4056a45596bddfb301f6dd5867c22a355a4c8d44d4c0052d\": rpc error: code = NotFound desc = could not find container \"8cd02ec096fa460a4056a45596bddfb301f6dd5867c22a355a4c8d44d4c0052d\": container with ID starting with 8cd02ec096fa460a4056a45596bddfb301f6dd5867c22a355a4c8d44d4c0052d not found: ID does not exist"
Nov 03 20:31:11.746500 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.746473    4421 scope.go:122] "RemoveContainer" containerID="2884e4fab020ea09b5e6f5d307306ff6e6778c87b614512d886f9c67b009a517"
Nov 03 20:31:11.746807 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:31:11.746786    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2884e4fab020ea09b5e6f5d307306ff6e6778c87b614512d886f9c67b009a517\": container with ID starting with 2884e4fab020ea09b5e6f5d307306ff6e6778c87b614512d886f9c67b009a517 not found: ID does not exist" containerID="2884e4fab020ea09b5e6f5d307306ff6e6778c87b614512d886f9c67b009a517"
Nov 03 20:31:11.746870 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.746816    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2884e4fab020ea09b5e6f5d307306ff6e6778c87b614512d886f9c67b009a517"} err="failed to get container status \"2884e4fab020ea09b5e6f5d307306ff6e6778c87b614512d886f9c67b009a517\": rpc error: code = NotFound desc = could not find container \"2884e4fab020ea09b5e6f5d307306ff6e6778c87b614512d886f9c67b009a517\": container with ID starting with 2884e4fab020ea09b5e6f5d307306ff6e6778c87b614512d886f9c67b009a517 not found: ID does not exist"
Nov 03 20:31:11.921359 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:11.921332    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:31:12.036814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.025205    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-workload-socket\") pod \"484992e4-0e7a-4728-9adb-0106b20d4dc0\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") "
Nov 03 20:31:12.036814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.025301    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-podinfo\") pod \"484992e4-0e7a-4728-9adb-0106b20d4dc0\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") "
Nov 03 20:31:12.036814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.025357    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/484992e4-0e7a-4728-9adb-0106b20d4dc0-kube-api-access-42p5g\" (UniqueName: \"kubernetes.io/projected/484992e4-0e7a-4728-9adb-0106b20d4dc0-kube-api-access-42p5g\") pod \"484992e4-0e7a-4728-9adb-0106b20d4dc0\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") "
Nov 03 20:31:12.036814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.025528    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-envoy\") pod \"484992e4-0e7a-4728-9adb-0106b20d4dc0\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") "
Nov 03 20:31:12.036814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.025639    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-token\" (UniqueName: \"kubernetes.io/projected/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-token\") pod \"484992e4-0e7a-4728-9adb-0106b20d4dc0\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") "
Nov 03 20:31:12.036814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.025684    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-workload-certs\") pod \"484992e4-0e7a-4728-9adb-0106b20d4dc0\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") "
Nov 03 20:31:12.036814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.025736    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/484992e4-0e7a-4728-9adb-0106b20d4dc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484992e4-0e7a-4728-9adb-0106b20d4dc0-istiod-ca-cert\") pod \"484992e4-0e7a-4728-9adb-0106b20d4dc0\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") "
Nov 03 20:31:12.036814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.025773    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-credential-socket\") pod \"484992e4-0e7a-4728-9adb-0106b20d4dc0\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") "
Nov 03 20:31:12.036814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.025892    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-data\") pod \"484992e4-0e7a-4728-9adb-0106b20d4dc0\" (UID: \"484992e4-0e7a-4728-9adb-0106b20d4dc0\") "
Nov 03 20:31:12.036814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.026392    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-workload-certs" pod "484992e4-0e7a-4728-9adb-0106b20d4dc0" (UID: "484992e4-0e7a-4728-9adb-0106b20d4dc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:12.036814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.026929    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/484992e4-0e7a-4728-9adb-0106b20d4dc0-istiod-ca-cert" pod "484992e4-0e7a-4728-9adb-0106b20d4dc0" (UID: "484992e4-0e7a-4728-9adb-0106b20d4dc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:12.036814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.027095    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:12.036814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.027144    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484992e4-0e7a-4728-9adb-0106b20d4dc0-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:12.036814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.027542    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-data" pod "484992e4-0e7a-4728-9adb-0106b20d4dc0" (UID: "484992e4-0e7a-4728-9adb-0106b20d4dc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:12.036814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.027841    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-credential-socket" pod "484992e4-0e7a-4728-9adb-0106b20d4dc0" (UID: "484992e4-0e7a-4728-9adb-0106b20d4dc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:12.036814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.028434    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484992e4-0e7a-4728-9adb-0106b20d4dc0-kube-api-access-42p5g" pod "484992e4-0e7a-4728-9adb-0106b20d4dc0" (UID: "484992e4-0e7a-4728-9adb-0106b20d4dc0"). InnerVolumeSpecName "kube-api-access-42p5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:12.036814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.028557    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-podinfo" pod "484992e4-0e7a-4728-9adb-0106b20d4dc0" (UID: "484992e4-0e7a-4728-9adb-0106b20d4dc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:31:12.036814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.028728    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-workload-socket" pod "484992e4-0e7a-4728-9adb-0106b20d4dc0" (UID: "484992e4-0e7a-4728-9adb-0106b20d4dc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:12.036814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.032471    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b270c5c6-0444-4410-a70d-37edb799adce" path="/var/lib/kubelet/pods/b270c5c6-0444-4410-a70d-37edb799adce/volumes"
Nov 03 20:31:12.036814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.033790    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-envoy" pod "484992e4-0e7a-4728-9adb-0106b20d4dc0" (UID: "484992e4-0e7a-4728-9adb-0106b20d4dc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:12.036814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.035770    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-token" pod "484992e4-0e7a-4728-9adb-0106b20d4dc0" (UID: "484992e4-0e7a-4728-9adb-0106b20d4dc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:12.128748 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.128717    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:12.128748 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.128746    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:12.128961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.128761    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42p5g\" (UniqueName: \"kubernetes.io/projected/484992e4-0e7a-4728-9adb-0106b20d4dc0-kube-api-access-42p5g\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:12.128961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.128773    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:12.128961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.128785    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:12.128961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.128796    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:12.128961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.128809    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484992e4-0e7a-4728-9adb-0106b20d4dc0-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:12.671765 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.671730    4421 generic.go:356] "Generic (PLEG): container finished" podID="484992e4-0e7a-4728-9adb-0106b20d4dc0" containerID="156e81e44712c985e6dde2291cecf41fc8ca94fe9917ea0df651460cfac77979" exitCode=0
Nov 03 20:31:12.672180 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.671785    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"
Nov 03 20:31:12.672180 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.671823    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx" event={"ID":"484992e4-0e7a-4728-9adb-0106b20d4dc0","Type":"ContainerDied","Data":"156e81e44712c985e6dde2291cecf41fc8ca94fe9917ea0df651460cfac77979"}
Nov 03 20:31:12.672180 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.671879    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx" event={"ID":"484992e4-0e7a-4728-9adb-0106b20d4dc0","Type":"ContainerDied","Data":"c591f062e3011f31bd21408dc8c475b542cacc0c2eeaa915c33657afd354815b"}
Nov 03 20:31:12.672180 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.671903    4421 scope.go:122] "RemoveContainer" containerID="156e81e44712c985e6dde2291cecf41fc8ca94fe9917ea0df651460cfac77979"
Nov 03 20:31:12.683932 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.683910    4421 scope.go:122] "RemoveContainer" containerID="156e81e44712c985e6dde2291cecf41fc8ca94fe9917ea0df651460cfac77979"
Nov 03 20:31:12.684234 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:31:12.684209    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"156e81e44712c985e6dde2291cecf41fc8ca94fe9917ea0df651460cfac77979\": container with ID starting with 156e81e44712c985e6dde2291cecf41fc8ca94fe9917ea0df651460cfac77979 not found: ID does not exist" containerID="156e81e44712c985e6dde2291cecf41fc8ca94fe9917ea0df651460cfac77979"
Nov 03 20:31:12.684313 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.684245    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"156e81e44712c985e6dde2291cecf41fc8ca94fe9917ea0df651460cfac77979"} err="failed to get container status \"156e81e44712c985e6dde2291cecf41fc8ca94fe9917ea0df651460cfac77979\": rpc error: code = NotFound desc = could not find container \"156e81e44712c985e6dde2291cecf41fc8ca94fe9917ea0df651460cfac77979\": container with ID starting with 156e81e44712c985e6dde2291cecf41fc8ca94fe9917ea0df651460cfac77979 not found: ID does not exist"
Nov 03 20:31:12.698963 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.698940    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"]
Nov 03 20:31:12.703500 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:12.703472    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-zrpdx"]
Nov 03 20:31:14.019241 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:14.019207    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="484992e4-0e7a-4728-9adb-0106b20d4dc0" path="/var/lib/kubelet/pods/484992e4-0e7a-4728-9adb-0106b20d4dc0/volumes"
Nov 03 20:31:14.951975 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:14.951936    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq"]
Nov 03 20:31:19.630814 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:31:19.630776    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:31:19.952904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:19.952868    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq" podUID="f497057c-a42e-452f-87f1-f4d197459616" containerName="payload-processing" containerID="cri-o://cfa22993e7b50bcf76fe4fb0e3e0a25605d81f3b64aa88a1b5d9d103c442aa58" gracePeriod=25
Nov 03 20:31:25.982941 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:25.982899    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq" podUID="f497057c-a42e-452f-87f1-f4d197459616" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:31:35.983492 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:35.983449    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq" podUID="f497057c-a42e-452f-87f1-f4d197459616" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:31:39.073851 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:39.073815    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2"]
Nov 03 20:31:39.074470 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:39.074080    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2" podUID="bac7a1a2-4692-4b09-9bb8-1d3faaa02692" containerName="maas-api" containerID="cri-o://b626f0f0a542bb50969d3a88f5ab990fbb79fad6f7f36cb75f5e282f7fa0c728" gracePeriod=30
Nov 03 20:31:39.200070 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:39.200041    4421 generic.go:356] "Generic (PLEG): container finished" podID="bac7a1a2-4692-4b09-9bb8-1d3faaa02692" containerID="b626f0f0a542bb50969d3a88f5ab990fbb79fad6f7f36cb75f5e282f7fa0c728" exitCode=0
Nov 03 20:31:39.200244 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:39.200112    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2" event={"ID":"bac7a1a2-4692-4b09-9bb8-1d3faaa02692","Type":"ContainerDied","Data":"b626f0f0a542bb50969d3a88f5ab990fbb79fad6f7f36cb75f5e282f7fa0c728"}
Nov 03 20:31:39.772080 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:39.772048    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2"
Nov 03 20:31:39.783165 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:39.783132    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-maas-api-tls\") pod \"bac7a1a2-4692-4b09-9bb8-1d3faaa02692\" (UID: \"bac7a1a2-4692-4b09-9bb8-1d3faaa02692\") "
Nov 03 20:31:39.783343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:39.783230    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-openshift-service-ca\") pod \"bac7a1a2-4692-4b09-9bb8-1d3faaa02692\" (UID: \"bac7a1a2-4692-4b09-9bb8-1d3faaa02692\") "
Nov 03 20:31:39.783343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:39.783303    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-kube-api-access-5cbs2\" (UniqueName: \"kubernetes.io/projected/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-kube-api-access-5cbs2\") pod \"bac7a1a2-4692-4b09-9bb8-1d3faaa02692\" (UID: \"bac7a1a2-4692-4b09-9bb8-1d3faaa02692\") "
Nov 03 20:31:39.783664 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:39.783592    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-metrics-certs\") pod \"bac7a1a2-4692-4b09-9bb8-1d3faaa02692\" (UID: \"bac7a1a2-4692-4b09-9bb8-1d3faaa02692\") "
Nov 03 20:31:39.783956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:39.783929    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-openshift-service-ca" pod "bac7a1a2-4692-4b09-9bb8-1d3faaa02692" (UID: "bac7a1a2-4692-4b09-9bb8-1d3faaa02692"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:39.784540 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:39.784514    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:39.805635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:39.789167    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-kube-api-access-5cbs2" pod "bac7a1a2-4692-4b09-9bb8-1d3faaa02692" (UID: "bac7a1a2-4692-4b09-9bb8-1d3faaa02692"). InnerVolumeSpecName "kube-api-access-5cbs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:39.805635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:39.800981    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-maas-api-tls" pod "bac7a1a2-4692-4b09-9bb8-1d3faaa02692" (UID: "bac7a1a2-4692-4b09-9bb8-1d3faaa02692"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:39.809672 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:39.807439    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-metrics-certs" pod "bac7a1a2-4692-4b09-9bb8-1d3faaa02692" (UID: "bac7a1a2-4692-4b09-9bb8-1d3faaa02692"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:39.885676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:39.885639    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:39.885676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:39.885670    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cbs2\" (UniqueName: \"kubernetes.io/projected/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-kube-api-access-5cbs2\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:39.885676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:39.885684    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bac7a1a2-4692-4b09-9bb8-1d3faaa02692-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:40.213201 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:40.213171    4421 generic.go:356] "Generic (PLEG): container finished" podID="f497057c-a42e-452f-87f1-f4d197459616" containerID="cfa22993e7b50bcf76fe4fb0e3e0a25605d81f3b64aa88a1b5d9d103c442aa58" exitCode=0
Nov 03 20:31:40.213588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:40.213224    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq" event={"ID":"f497057c-a42e-452f-87f1-f4d197459616","Type":"ContainerDied","Data":"cfa22993e7b50bcf76fe4fb0e3e0a25605d81f3b64aa88a1b5d9d103c442aa58"}
Nov 03 20:31:40.215165 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:40.215124    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2"
Nov 03 20:31:40.215165 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:40.215127    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2" event={"ID":"bac7a1a2-4692-4b09-9bb8-1d3faaa02692","Type":"ContainerDied","Data":"9831ca3c71c1af8e55f9b4773cf052c613a7fab88cb7c591761a6860604152c1"}
Nov 03 20:31:40.215393 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:40.215184    4421 scope.go:122] "RemoveContainer" containerID="b626f0f0a542bb50969d3a88f5ab990fbb79fad6f7f36cb75f5e282f7fa0c728"
Nov 03 20:31:40.237686 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:40.237655    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2"]
Nov 03 20:31:40.243881 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:40.243859    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3211a0-59f658c764-55vv2"]
Nov 03 20:31:40.278336 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:40.278314    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq"
Nov 03 20:31:40.292896 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:40.292867    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f497057c-a42e-452f-87f1-f4d197459616-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f497057c-a42e-452f-87f1-f4d197459616-plugins-config-volume\") pod \"f497057c-a42e-452f-87f1-f4d197459616\" (UID: \"f497057c-a42e-452f-87f1-f4d197459616\") "
Nov 03 20:31:40.293143 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:40.293127    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f497057c-a42e-452f-87f1-f4d197459616-kube-api-access-x5742\" (UniqueName: \"kubernetes.io/projected/f497057c-a42e-452f-87f1-f4d197459616-kube-api-access-x5742\") pod \"f497057c-a42e-452f-87f1-f4d197459616\" (UID: \"f497057c-a42e-452f-87f1-f4d197459616\") "
Nov 03 20:31:40.293246 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:40.293222    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f497057c-a42e-452f-87f1-f4d197459616-plugins-config-volume" pod "f497057c-a42e-452f-87f1-f4d197459616" (UID: "f497057c-a42e-452f-87f1-f4d197459616"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:40.293756 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:40.293736    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f497057c-a42e-452f-87f1-f4d197459616-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:40.295408 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:40.295386    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f497057c-a42e-452f-87f1-f4d197459616-kube-api-access-x5742" pod "f497057c-a42e-452f-87f1-f4d197459616" (UID: "f497057c-a42e-452f-87f1-f4d197459616"). InnerVolumeSpecName "kube-api-access-x5742". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:40.395289 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:40.395249    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5742\" (UniqueName: \"kubernetes.io/projected/f497057c-a42e-452f-87f1-f4d197459616-kube-api-access-x5742\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:41.315032 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:41.314989    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq" event={"ID":"f497057c-a42e-452f-87f1-f4d197459616","Type":"ContainerDied","Data":"bcf03279203542401acc1e4edaff6d6a772f93ad78687fd07d30d8f51d1c01b7"}
Nov 03 20:31:41.315515 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:41.315064    4421 scope.go:122] "RemoveContainer" containerID="cfa22993e7b50bcf76fe4fb0e3e0a25605d81f3b64aa88a1b5d9d103c442aa58"
Nov 03 20:31:41.315515 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:41.315224    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq"
Nov 03 20:31:41.357502 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:41.357472    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq"]
Nov 03 20:31:41.362009 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:41.361986    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3211a0-646899f4fb-j2thq"]
Nov 03 20:31:42.097442 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:42.097403    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bac7a1a2-4692-4b09-9bb8-1d3faaa02692" path="/var/lib/kubelet/pods/bac7a1a2-4692-4b09-9bb8-1d3faaa02692/volumes"
Nov 03 20:31:42.098393 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:42.098357    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f497057c-a42e-452f-87f1-f4d197459616" path="/var/lib/kubelet/pods/f497057c-a42e-452f-87f1-f4d197459616/volumes"
Nov 03 20:31:44.509903 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:44.508762    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"]
Nov 03 20:31:44.509903 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:44.509048    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7" podUID="2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511" containerName="istio-proxy" containerID="cri-o://a16db7da3590f2b747019ce5b5365908f90124e9ec3f444441cf374abe373047" gracePeriod=30
Nov 03 20:31:44.515590 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:44.515563    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"]
Nov 03 20:31:44.515892 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:44.515829    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69" podUID="d6bfef45-f4d7-4482-bde4-2328d5e54b6c" containerName="istio-proxy" containerID="cri-o://b62b11e0268b8b598c2358b21336c2c65c6b0b5ad6b08ab13b1e33daf4a7e002" gracePeriod=30
Nov 03 20:31:44.520568 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:44.520547    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"]
Nov 03 20:31:44.520814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:44.520785    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw" podUID="47d17b9b-ecc0-471b-ae3e-30b76b7241ce" containerName="istio-proxy" containerID="cri-o://c9ea841ef292767c409a7cd168fb4615091160264bf906dee439a0747dc0d842" gracePeriod=30
Nov 03 20:31:45.720592 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:45.720562    4421 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:31:45.721047 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:45.720646    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69" podUID="d6bfef45-f4d7-4482-bde4-2328d5e54b6c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:31:49.524088 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.524013    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"]
Nov 03 20:31:49.524923 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.524906    4421 cpu_manager.go:414] "containerMap: removing container" podUID="b270c5c6-0444-4410-a70d-37edb799adce" containerName="istio-proxy"
Nov 03 20:31:49.524983 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.524925    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b270c5c6-0444-4410-a70d-37edb799adce" containerName="istio-proxy"
Nov 03 20:31:49.524983 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.524937    4421 cpu_manager.go:414] "containerMap: removing container" podUID="484992e4-0e7a-4728-9adb-0106b20d4dc0" containerName="istio-proxy"
Nov 03 20:31:49.524983 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.524942    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="484992e4-0e7a-4728-9adb-0106b20d4dc0" containerName="istio-proxy"
Nov 03 20:31:49.524983 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.524965    4421 cpu_manager.go:414] "containerMap: removing container" podUID="bac7a1a2-4692-4b09-9bb8-1d3faaa02692" containerName="maas-api"
Nov 03 20:31:49.524983 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.524970    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac7a1a2-4692-4b09-9bb8-1d3faaa02692" containerName="maas-api"
Nov 03 20:31:49.524983 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.524983    4421 cpu_manager.go:414] "containerMap: removing container" podUID="2f1901a6-6aeb-4786-88a7-1b7836495531" containerName="cleanup"
Nov 03 20:31:49.525185 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.524987    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1901a6-6aeb-4786-88a7-1b7836495531" containerName="cleanup"
Nov 03 20:31:49.525185 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.524999    4421 cpu_manager.go:414] "containerMap: removing container" podUID="f497057c-a42e-452f-87f1-f4d197459616" containerName="payload-processing"
Nov 03 20:31:49.525185 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.525003    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f497057c-a42e-452f-87f1-f4d197459616" containerName="payload-processing"
Nov 03 20:31:49.525185 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.525136    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="b270c5c6-0444-4410-a70d-37edb799adce" containerName="istio-proxy"
Nov 03 20:31:49.525185 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.525147    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f497057c-a42e-452f-87f1-f4d197459616" containerName="payload-processing"
Nov 03 20:31:49.525185 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.525156    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1901a6-6aeb-4786-88a7-1b7836495531" containerName="cleanup"
Nov 03 20:31:49.525185 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.525165    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="484992e4-0e7a-4728-9adb-0106b20d4dc0" containerName="istio-proxy"
Nov 03 20:31:49.525185 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.525173    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f785b3f-e95b-4a67-b7f8-4e1731a7e070" containerName="cleanup"
Nov 03 20:31:49.525185 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.525179    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac7a1a2-4692-4b09-9bb8-1d3faaa02692" containerName="maas-api"
Nov 03 20:31:49.530827 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.530803    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.537379 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.537356    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-99a8fe-openshift-default-dockercfg-5wmrx\""
Nov 03 20:31:49.541282 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.541255    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"]
Nov 03 20:31:49.598894 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.598865    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-token\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.599014 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.598922    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-envoy\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.599063 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.599012    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-data\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.599214 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.599194    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-workload-socket\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.599264 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.599223    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istiod-ca-cert\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.599264 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.599259    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxpbs\" (UniqueName: \"kubernetes.io/projected/32697d8f-a748-4deb-8e4c-5c8a08c235f4-kube-api-access-kxpbs\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.599392 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.599374    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-credential-socket\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.599442 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.599421    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-podinfo\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.599489 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.599440    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-workload-certs\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.646381 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:31:49.646338    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:31:49.701111 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.701059    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-workload-socket\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.701111 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.701101    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istiod-ca-cert\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.701393 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.701138    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxpbs\" (UniqueName: \"kubernetes.io/projected/32697d8f-a748-4deb-8e4c-5c8a08c235f4-kube-api-access-kxpbs\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.701393 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.701195    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-credential-socket\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.701393 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.701241    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-podinfo\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.701393 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.701261    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-workload-certs\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.701393 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.701357    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-token\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.701657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.701398    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-envoy\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.701657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.701443    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-data\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.702335 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.701992    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-data\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.702335 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.702050    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-credential-socket\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.702335 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.702082    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-workload-certs\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.702587 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.702557    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-workload-socket\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.703335 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.702732    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istiod-ca-cert\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.705660 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.705064    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-envoy\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.705812 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.705794    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-podinfo\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.715530 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.715503    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-token\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.737464 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.718985    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxpbs\" (UniqueName: \"kubernetes.io/projected/32697d8f-a748-4deb-8e4c-5c8a08c235f4-kube-api-access-kxpbs\") pod \"e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") " pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.857989 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.857892    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:49.919080 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:49.918732    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:31:50.022641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.020628    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-token\" (UniqueName: \"kubernetes.io/projected/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-token\") pod \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") "
Nov 03 20:31:50.022641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.020720    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-podinfo\") pod \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") "
Nov 03 20:31:50.022641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.020899    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-envoy\") pod \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") "
Nov 03 20:31:50.022641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.020943    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-workload-socket\") pod \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") "
Nov 03 20:31:50.022641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.020968    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istiod-ca-cert\") pod \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") "
Nov 03 20:31:50.022641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.021050    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-credential-socket\") pod \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") "
Nov 03 20:31:50.022641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.021176    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-workload-certs\") pod \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") "
Nov 03 20:31:50.022641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.021255    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-kube-api-access-tjkv5\" (UniqueName: \"kubernetes.io/projected/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-kube-api-access-tjkv5\") pod \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") "
Nov 03 20:31:50.022641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.021327    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-data\") pod \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\" (UID: \"d6bfef45-f4d7-4482-bde4-2328d5e54b6c\") "
Nov 03 20:31:50.027893 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.022976    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-credential-socket" pod "d6bfef45-f4d7-4482-bde4-2328d5e54b6c" (UID: "d6bfef45-f4d7-4482-bde4-2328d5e54b6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:50.027893 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.024567    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-workload-socket" pod "d6bfef45-f4d7-4482-bde4-2328d5e54b6c" (UID: "d6bfef45-f4d7-4482-bde4-2328d5e54b6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:50.027893 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.027287    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istiod-ca-cert" pod "d6bfef45-f4d7-4482-bde4-2328d5e54b6c" (UID: "d6bfef45-f4d7-4482-bde4-2328d5e54b6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:50.031799 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.029570    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-token" pod "d6bfef45-f4d7-4482-bde4-2328d5e54b6c" (UID: "d6bfef45-f4d7-4482-bde4-2328d5e54b6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:50.033032 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.032989    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-workload-certs" pod "d6bfef45-f4d7-4482-bde4-2328d5e54b6c" (UID: "d6bfef45-f4d7-4482-bde4-2328d5e54b6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:50.039119 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.038972    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-kube-api-access-tjkv5" pod "d6bfef45-f4d7-4482-bde4-2328d5e54b6c" (UID: "d6bfef45-f4d7-4482-bde4-2328d5e54b6c"). InnerVolumeSpecName "kube-api-access-tjkv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:50.040138 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.039748    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-data" pod "d6bfef45-f4d7-4482-bde4-2328d5e54b6c" (UID: "d6bfef45-f4d7-4482-bde4-2328d5e54b6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:50.043133 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.043106    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-envoy" pod "d6bfef45-f4d7-4482-bde4-2328d5e54b6c" (UID: "d6bfef45-f4d7-4482-bde4-2328d5e54b6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:50.044578 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.044510    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:31:50.047202 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.047156    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-podinfo" pod "d6bfef45-f4d7-4482-bde4-2328d5e54b6c" (UID: "d6bfef45-f4d7-4482-bde4-2328d5e54b6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:31:50.050390 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.049825    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:31:50.123288 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.123201    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-credential-socket\") pod \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") "
Nov 03 20:31:50.123462 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.123313    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-workload-socket\") pod \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") "
Nov 03 20:31:50.123462 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.123357    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-token\" (UniqueName: \"kubernetes.io/projected/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-token\") pod \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") "
Nov 03 20:31:50.123462 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.123384    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-workload-socket\") pod \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") "
Nov 03 20:31:50.123462 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.123418    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-workload-certs\") pod \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") "
Nov 03 20:31:50.123462 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.123460    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-data\") pod \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") "
Nov 03 20:31:50.123788 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.123494    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-podinfo\") pod \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") "
Nov 03 20:31:50.123788 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.123524    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-credential-socket" pod "2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511" (UID: "2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:50.123788 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.123584    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-kube-api-access-gbk9l\" (UniqueName: \"kubernetes.io/projected/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-kube-api-access-gbk9l\") pod \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") "
Nov 03 20:31:50.123788 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.123672    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-workload-socket" pod "47d17b9b-ecc0-471b-ae3e-30b76b7241ce" (UID: "47d17b9b-ecc0-471b-ae3e-30b76b7241ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:50.123788 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.123704    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-token\" (UniqueName: \"kubernetes.io/projected/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-token\") pod \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") "
Nov 03 20:31:50.124036 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.123789    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-data" pod "47d17b9b-ecc0-471b-ae3e-30b76b7241ce" (UID: "47d17b9b-ecc0-471b-ae3e-30b76b7241ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:50.124036 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.123850    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-envoy\") pod \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") "
Nov 03 20:31:50.124036 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.123918    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istiod-ca-cert\") pod \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") "
Nov 03 20:31:50.124036 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.123938    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-workload-certs\") pod \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") "
Nov 03 20:31:50.124036 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.123967    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-data\") pod \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") "
Nov 03 20:31:50.124036 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.124029    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istiod-ca-cert\") pod \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") "
Nov 03 20:31:50.124340 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.124093    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-envoy\") pod \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") "
Nov 03 20:31:50.124340 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.124121    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-credential-socket\") pod \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") "
Nov 03 20:31:50.124340 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.124178    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-kube-api-access-4f788\" (UniqueName: \"kubernetes.io/projected/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-kube-api-access-4f788\") pod \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\" (UID: \"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511\") "
Nov 03 20:31:50.124340 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.124209    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-podinfo\") pod \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\" (UID: \"47d17b9b-ecc0-471b-ae3e-30b76b7241ce\") "
Nov 03 20:31:50.124538 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.124369    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-workload-certs" pod "47d17b9b-ecc0-471b-ae3e-30b76b7241ce" (UID: "47d17b9b-ecc0-471b-ae3e-30b76b7241ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:50.124635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.124592    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-workload-certs" pod "2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511" (UID: "2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:50.124904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.124883    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-workload-socket" pod "2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511" (UID: "2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:50.125248 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.125226    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istiod-ca-cert" pod "47d17b9b-ecc0-471b-ae3e-30b76b7241ce" (UID: "47d17b9b-ecc0-471b-ae3e-30b76b7241ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:50.125470 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.125452    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-data" pod "2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511" (UID: "2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:50.125721 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.125703    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istiod-ca-cert" pod "2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511" (UID: "2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:50.126165 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.126147    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.126263 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.126170    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.126263 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.126185    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjkv5\" (UniqueName: \"kubernetes.io/projected/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-kube-api-access-tjkv5\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.126263 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.126199    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.126263 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.126211    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.126263 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.126224    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.126263 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.126237    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.126263 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.126251    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.126263 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.126264    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.126755 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.126277    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.126755 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.126290    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.126755 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.126304    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.126755 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.126318    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.126755 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.126332    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.126755 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.126346    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6bfef45-f4d7-4482-bde4-2328d5e54b6c-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.126755 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.126361    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.126755 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.126375    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.126755 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.126389    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.126755 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.126660    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-credential-socket" pod "47d17b9b-ecc0-471b-ae3e-30b76b7241ce" (UID: "47d17b9b-ecc0-471b-ae3e-30b76b7241ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:50.127572 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.127549    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-envoy" pod "2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511" (UID: "2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:50.129656 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.129557    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-kube-api-access-gbk9l" pod "47d17b9b-ecc0-471b-ae3e-30b76b7241ce" (UID: "47d17b9b-ecc0-471b-ae3e-30b76b7241ce"). InnerVolumeSpecName "kube-api-access-gbk9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:50.129920 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.129900    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-podinfo" pod "2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511" (UID: "2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:31:50.132578 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.131073    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-token" pod "2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511" (UID: "2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:50.132815 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.131972    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-kube-api-access-4f788" pod "2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511" (UID: "2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511"). InnerVolumeSpecName "kube-api-access-4f788". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:50.132815 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.132320    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-envoy" pod "47d17b9b-ecc0-471b-ae3e-30b76b7241ce" (UID: "47d17b9b-ecc0-471b-ae3e-30b76b7241ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:50.132962 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.132911    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-token" pod "47d17b9b-ecc0-471b-ae3e-30b76b7241ce" (UID: "47d17b9b-ecc0-471b-ae3e-30b76b7241ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:50.133132 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.133107    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-podinfo" pod "47d17b9b-ecc0-471b-ae3e-30b76b7241ce" (UID: "47d17b9b-ecc0-471b-ae3e-30b76b7241ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:31:50.172432 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.172403    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"]
Nov 03 20:31:50.172751 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:31:50.172726    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32697d8f_a748_4deb_8e4c_5c8a08c235f4.slice/crio-76c7d3b37e3660dd067d5ec737cc9b837bd85ba31988ab0b5521648a6ff45bec WatchSource:0}: Error finding container 76c7d3b37e3660dd067d5ec737cc9b837bd85ba31988ab0b5521648a6ff45bec: Status 404 returned error can't find the container with id 76c7d3b37e3660dd067d5ec737cc9b837bd85ba31988ab0b5521648a6ff45bec
Nov 03 20:31:50.176300 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.176253    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:31:50.176388 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.176353    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:31:50.176437 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.176399    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:31:50.227874 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.227842    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.227874 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.227873    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.228099 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.227884    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbk9l\" (UniqueName: \"kubernetes.io/projected/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-kube-api-access-gbk9l\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.228099 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.227896    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.228099 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.227905    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.228099 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.227913    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.228099 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.227922    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.228099 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.227930    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f788\" (UniqueName: \"kubernetes.io/projected/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511-kube-api-access-4f788\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.228099 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.227944    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d17b9b-ecc0-471b-ae3e-30b76b7241ce-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:50.406780 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.406693    4421 generic.go:356] "Generic (PLEG): container finished" podID="2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511" containerID="a16db7da3590f2b747019ce5b5365908f90124e9ec3f444441cf374abe373047" exitCode=0
Nov 03 20:31:50.406780 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.406742    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7" event={"ID":"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511","Type":"ContainerDied","Data":"a16db7da3590f2b747019ce5b5365908f90124e9ec3f444441cf374abe373047"}
Nov 03 20:31:50.406780 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.406769    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"
Nov 03 20:31:50.407052 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.406803    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7" event={"ID":"2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511","Type":"ContainerDied","Data":"68c8ef004c764f91684ccdd253872c2feb8a540a9772b92c440dc2d826abddb2"}
Nov 03 20:31:50.407052 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.406827    4421 scope.go:122] "RemoveContainer" containerID="a16db7da3590f2b747019ce5b5365908f90124e9ec3f444441cf374abe373047"
Nov 03 20:31:50.409713 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.409689    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf" event={"ID":"32697d8f-a748-4deb-8e4c-5c8a08c235f4","Type":"ContainerStarted","Data":"900d2c38acb4a4b584714b8f48c7e4b38bacec834cca965e46de926539afacf1"}
Nov 03 20:31:50.409859 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.409736    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf" event={"ID":"32697d8f-a748-4deb-8e4c-5c8a08c235f4","Type":"ContainerStarted","Data":"76c7d3b37e3660dd067d5ec737cc9b837bd85ba31988ab0b5521648a6ff45bec"}
Nov 03 20:31:50.411388 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.411362    4421 generic.go:356] "Generic (PLEG): container finished" podID="47d17b9b-ecc0-471b-ae3e-30b76b7241ce" containerID="c9ea841ef292767c409a7cd168fb4615091160264bf906dee439a0747dc0d842" exitCode=0
Nov 03 20:31:50.411486 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.411420    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw" event={"ID":"47d17b9b-ecc0-471b-ae3e-30b76b7241ce","Type":"ContainerDied","Data":"c9ea841ef292767c409a7cd168fb4615091160264bf906dee439a0747dc0d842"}
Nov 03 20:31:50.411486 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.411436    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"
Nov 03 20:31:50.411486 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.411470    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw" event={"ID":"47d17b9b-ecc0-471b-ae3e-30b76b7241ce","Type":"ContainerDied","Data":"bc505efacb314a2351c428a09aec68618627948bf8a5aaf0cc362c3310e770e9"}
Nov 03 20:31:50.413227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.413202    4421 generic.go:356] "Generic (PLEG): container finished" podID="d6bfef45-f4d7-4482-bde4-2328d5e54b6c" containerID="b62b11e0268b8b598c2358b21336c2c65c6b0b5ad6b08ab13b1e33daf4a7e002" exitCode=0
Nov 03 20:31:50.413344 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.413250    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"
Nov 03 20:31:50.413344 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.413289    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69" event={"ID":"d6bfef45-f4d7-4482-bde4-2328d5e54b6c","Type":"ContainerDied","Data":"b62b11e0268b8b598c2358b21336c2c65c6b0b5ad6b08ab13b1e33daf4a7e002"}
Nov 03 20:31:50.413488 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.413369    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69" event={"ID":"d6bfef45-f4d7-4482-bde4-2328d5e54b6c","Type":"ContainerDied","Data":"92068db831ba35380885d023f4660762f11da1800947f47c0915dcac8035ba5b"}
Nov 03 20:31:50.423556 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.423535    4421 scope.go:122] "RemoveContainer" containerID="a16db7da3590f2b747019ce5b5365908f90124e9ec3f444441cf374abe373047"
Nov 03 20:31:50.423842 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:31:50.423824    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a16db7da3590f2b747019ce5b5365908f90124e9ec3f444441cf374abe373047\": container with ID starting with a16db7da3590f2b747019ce5b5365908f90124e9ec3f444441cf374abe373047 not found: ID does not exist" containerID="a16db7da3590f2b747019ce5b5365908f90124e9ec3f444441cf374abe373047"
Nov 03 20:31:50.423977 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.423855    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a16db7da3590f2b747019ce5b5365908f90124e9ec3f444441cf374abe373047"} err="failed to get container status \"a16db7da3590f2b747019ce5b5365908f90124e9ec3f444441cf374abe373047\": rpc error: code = NotFound desc = could not find container \"a16db7da3590f2b747019ce5b5365908f90124e9ec3f444441cf374abe373047\": container with ID starting with a16db7da3590f2b747019ce5b5365908f90124e9ec3f444441cf374abe373047 not found: ID does not exist"
Nov 03 20:31:50.423977 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.423880    4421 scope.go:122] "RemoveContainer" containerID="c9ea841ef292767c409a7cd168fb4615091160264bf906dee439a0747dc0d842"
Nov 03 20:31:50.436132 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.436093    4421 scope.go:122] "RemoveContainer" containerID="c9ea841ef292767c409a7cd168fb4615091160264bf906dee439a0747dc0d842"
Nov 03 20:31:50.436713 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:31:50.436686    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9ea841ef292767c409a7cd168fb4615091160264bf906dee439a0747dc0d842\": container with ID starting with c9ea841ef292767c409a7cd168fb4615091160264bf906dee439a0747dc0d842 not found: ID does not exist" containerID="c9ea841ef292767c409a7cd168fb4615091160264bf906dee439a0747dc0d842"
Nov 03 20:31:50.436820 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.436720    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9ea841ef292767c409a7cd168fb4615091160264bf906dee439a0747dc0d842"} err="failed to get container status \"c9ea841ef292767c409a7cd168fb4615091160264bf906dee439a0747dc0d842\": rpc error: code = NotFound desc = could not find container \"c9ea841ef292767c409a7cd168fb4615091160264bf906dee439a0747dc0d842\": container with ID starting with c9ea841ef292767c409a7cd168fb4615091160264bf906dee439a0747dc0d842 not found: ID does not exist"
Nov 03 20:31:50.436820 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.436742    4421 scope.go:122] "RemoveContainer" containerID="b62b11e0268b8b598c2358b21336c2c65c6b0b5ad6b08ab13b1e33daf4a7e002"
Nov 03 20:31:50.436986 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.436950    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf" podStartSLOduration=1.436939876 podStartE2EDuration="1.436939876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:50.43288296 +0000 UTC m=+1840.566011423" watchObservedRunningTime="2026-11-03 20:31:50.436939876 +0000 UTC m=+1840.570068337"
Nov 03 20:31:50.449084 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.449061    4421 scope.go:122] "RemoveContainer" containerID="b62b11e0268b8b598c2358b21336c2c65c6b0b5ad6b08ab13b1e33daf4a7e002"
Nov 03 20:31:50.449419 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:31:50.449390    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b62b11e0268b8b598c2358b21336c2c65c6b0b5ad6b08ab13b1e33daf4a7e002\": container with ID starting with b62b11e0268b8b598c2358b21336c2c65c6b0b5ad6b08ab13b1e33daf4a7e002 not found: ID does not exist" containerID="b62b11e0268b8b598c2358b21336c2c65c6b0b5ad6b08ab13b1e33daf4a7e002"
Nov 03 20:31:50.449543 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.449425    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b62b11e0268b8b598c2358b21336c2c65c6b0b5ad6b08ab13b1e33daf4a7e002"} err="failed to get container status \"b62b11e0268b8b598c2358b21336c2c65c6b0b5ad6b08ab13b1e33daf4a7e002\": rpc error: code = NotFound desc = could not find container \"b62b11e0268b8b598c2358b21336c2c65c6b0b5ad6b08ab13b1e33daf4a7e002\": container with ID starting with b62b11e0268b8b598c2358b21336c2c65c6b0b5ad6b08ab13b1e33daf4a7e002 not found: ID does not exist"
Nov 03 20:31:50.452111 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.452090    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"]
Nov 03 20:31:50.456667 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.456647    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-mtctw"]
Nov 03 20:31:50.467933 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.467910    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"]
Nov 03 20:31:50.473873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.473853    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-gfmt7"]
Nov 03 20:31:50.486643 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.486584    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"]
Nov 03 20:31:50.493298 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.493269    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-fbt69"]
Nov 03 20:31:50.863127 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:50.862699    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:51.084702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.084675    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"]
Nov 03 20:31:51.092651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.085635    4421 cpu_manager.go:414] "containerMap: removing container" podUID="d6bfef45-f4d7-4482-bde4-2328d5e54b6c" containerName="istio-proxy"
Nov 03 20:31:51.092651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.085659    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6bfef45-f4d7-4482-bde4-2328d5e54b6c" containerName="istio-proxy"
Nov 03 20:31:51.092651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.085693    4421 cpu_manager.go:414] "containerMap: removing container" podUID="47d17b9b-ecc0-471b-ae3e-30b76b7241ce" containerName="istio-proxy"
Nov 03 20:31:51.092651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.085699    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47d17b9b-ecc0-471b-ae3e-30b76b7241ce" containerName="istio-proxy"
Nov 03 20:31:51.092651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.085714    4421 cpu_manager.go:414] "containerMap: removing container" podUID="2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511" containerName="istio-proxy"
Nov 03 20:31:51.092651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.085720    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511" containerName="istio-proxy"
Nov 03 20:31:51.092651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.085740    4421 cpu_manager.go:414] "containerMap: removing container" podUID="b270c5c6-0444-4410-a70d-37edb799adce" containerName="istio-proxy"
Nov 03 20:31:51.092651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.085745    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b270c5c6-0444-4410-a70d-37edb799adce" containerName="istio-proxy"
Nov 03 20:31:51.092651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.085877    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511" containerName="istio-proxy"
Nov 03 20:31:51.092651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.085896    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6bfef45-f4d7-4482-bde4-2328d5e54b6c" containerName="istio-proxy"
Nov 03 20:31:51.092651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.085902    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="b270c5c6-0444-4410-a70d-37edb799adce" containerName="istio-proxy"
Nov 03 20:31:51.092651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.085913    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="47d17b9b-ecc0-471b-ae3e-30b76b7241ce" containerName="istio-proxy"
Nov 03 20:31:51.097626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.097230    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"]
Nov 03 20:31:51.097626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.097353    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.107136 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.107112    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-dce009-openshift-default-dockercfg-47pfw\""
Nov 03 20:31:51.455073 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.416761    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-token\") pod \"e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") " pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.455073 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.416836    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-envoy\") pod \"e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") " pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.455073 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.416904    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-credential-socket\") pod \"e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") " pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.455073 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.417050    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istiod-ca-cert\") pod \"e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") " pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.455073 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.417144    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-data\") pod \"e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") " pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.455073 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.417173    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-workload-certs\") pod \"e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") " pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.455073 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.417266    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-workload-socket\") pod \"e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") " pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.455073 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.417301    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2dk4\" (UniqueName: \"kubernetes.io/projected/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-kube-api-access-p2dk4\") pod \"e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") " pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.455073 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.417366    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-podinfo\") pod \"e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") " pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.560664 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.559612    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8"]
Nov 03 20:31:51.663625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.641040    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-token\") pod \"e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") " pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.663987 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.663955    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-envoy\") pod \"e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") " pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.664176 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.664158    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-credential-socket\") pod \"e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") " pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.664494 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.664468    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istiod-ca-cert\") pod \"e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") " pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.711278 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.686024    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-envoy\") pod \"e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") " pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.711491 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.686280    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-credential-socket\") pod \"e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") " pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.711593 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.687489    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istiod-ca-cert\") pod \"e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") " pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.711687 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.711312    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-data\") pod \"e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") " pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.711687 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.711672    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-data\") pod \"e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") " pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.711810 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.711761    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-workload-certs\") pod \"e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") " pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.712895 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.711893    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-workload-socket\") pod \"e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") " pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.712895 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.712228    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-workload-socket\") pod \"e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") " pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.712895 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.712270    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2dk4\" (UniqueName: \"kubernetes.io/projected/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-kube-api-access-p2dk4\") pod \"e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") " pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.712895 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.712328    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-podinfo\") pod \"e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") " pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.713879 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.713852    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-workload-certs\") pod \"e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") " pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.715251 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.714707    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-token\") pod \"e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") " pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.719677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.719651    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk"]
Nov 03 20:31:51.723626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.723234    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-podinfo\") pod \"e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") " pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.728160 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.728139    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2dk4\" (UniqueName: \"kubernetes.io/projected/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-kube-api-access-p2dk4\") pod \"e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") " pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:51.936624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.925533    4421 patch_prober.go:29] interesting pod/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:31:51.936624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:51.925630    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf" podUID="32697d8f-a748-4deb-8e4c-5c8a08c235f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Nov 03 20:31:52.016677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:52.016065    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:52.124704 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:52.124663    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511" path="/var/lib/kubelet/pods/2ba7fbba-9905-4eb1-b4e8-8c1ae6b0c511/volumes"
Nov 03 20:31:52.127871 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:52.127817    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47d17b9b-ecc0-471b-ae3e-30b76b7241ce" path="/var/lib/kubelet/pods/47d17b9b-ecc0-471b-ae3e-30b76b7241ce/volumes"
Nov 03 20:31:52.130748 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:52.130713    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6bfef45-f4d7-4482-bde4-2328d5e54b6c" path="/var/lib/kubelet/pods/d6bfef45-f4d7-4482-bde4-2328d5e54b6c/volumes"
Nov 03 20:31:52.514063 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:52.514036    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"]
Nov 03 20:31:52.514937 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:31:52.514910    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2fda22_a9be_4bf5_8b2b_0e994221c92b.slice/crio-a95ee4028f333c52a8e46d072e0a5b4de903e8773215d43021199dd5073e8fd1 WatchSource:0}: Error finding container a95ee4028f333c52a8e46d072e0a5b4de903e8773215d43021199dd5073e8fd1: Status 404 returned error can't find the container with id a95ee4028f333c52a8e46d072e0a5b4de903e8773215d43021199dd5073e8fd1
Nov 03 20:31:52.518129 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:52.518094    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:31:52.518230 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:52.518177    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:31:52.518230 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:52.518223    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:31:52.633643 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:52.633608    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j" event={"ID":"0a2fda22-a9be-4bf5-8b2b-0e994221c92b","Type":"ContainerStarted","Data":"6138acb76efaa00253d360733024c7f31ec60285410bbb53cd396b7f98c0917a"}
Nov 03 20:31:52.633795 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:52.633673    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j" event={"ID":"0a2fda22-a9be-4bf5-8b2b-0e994221c92b","Type":"ContainerStarted","Data":"a95ee4028f333c52a8e46d072e0a5b4de903e8773215d43021199dd5073e8fd1"}
Nov 03 20:31:52.863782 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:52.863669    4421 patch_prober.go:29] interesting pod/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:31:52.864068 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:52.864021    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf" podUID="32697d8f-a748-4deb-8e4c-5c8a08c235f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Nov 03 20:31:53.018179 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:53.018140    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:53.019370 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:53.019346    4421 patch_prober.go:29] interesting pod/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 03 20:31:53.019490 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:53.019393    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j" podUID="0a2fda22-a9be-4bf5-8b2b-0e994221c92b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 03 20:31:53.865624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:53.864737    4421 patch_prober.go:29] interesting pod/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:31:53.865624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:53.864796    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf" podUID="32697d8f-a748-4deb-8e4c-5c8a08c235f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Nov 03 20:31:54.061061 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:54.061032    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:54.094768 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:54.094715    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j" podStartSLOduration=3.094699091 podStartE2EDuration="3.094699091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:52.67468062 +0000 UTC m=+1842.807809106" watchObservedRunningTime="2026-11-03 20:31:54.094699091 +0000 UTC m=+1844.227827600"
Nov 03 20:31:54.697277 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:54.697246    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:54.700277 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:54.700258    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:31:54.871792 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:54.867686    4421 patch_prober.go:29] interesting pod/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:31:54.871792 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:54.867752    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf" podUID="32697d8f-a748-4deb-8e4c-5c8a08c235f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Nov 03 20:31:55.863622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:55.863105    4421 patch_prober.go:29] interesting pod/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:31:55.863622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:55.863166    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf" podUID="32697d8f-a748-4deb-8e4c-5c8a08c235f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Nov 03 20:31:56.184411 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:56.184110    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4"]
Nov 03 20:31:56.196163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:56.196136    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4"
Nov 03 20:31:56.208105 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:56.207845    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-dce009\""
Nov 03 20:31:56.208426 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:56.208407    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-dce009-dockercfg-kqglx\""
Nov 03 20:31:56.220547 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:56.217125    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4"]
Nov 03 20:31:56.438683 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:56.409576    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04160b7a-00d4-47d0-88dd-54646e94e9e5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4\" (UID: \"04160b7a-00d4-47d0-88dd-54646e94e9e5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4"
Nov 03 20:31:56.465488 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:56.465446    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w7bg\" (UniqueName: \"kubernetes.io/projected/04160b7a-00d4-47d0-88dd-54646e94e9e5-kube-api-access-7w7bg\") pod \"payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4\" (UID: \"04160b7a-00d4-47d0-88dd-54646e94e9e5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4"
Nov 03 20:31:56.562702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:56.562660    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8" podUID="fcfb2deb-19e8-4407-90f9-d20452299a86" containerName="payload-processing" containerID="cri-o://c4a98f9811a52e1181f604d28ead48a0b07858f624dff2e83b2fdc4408c8d625" gracePeriod=25
Nov 03 20:31:56.574622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:56.572496    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04160b7a-00d4-47d0-88dd-54646e94e9e5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4\" (UID: \"04160b7a-00d4-47d0-88dd-54646e94e9e5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4"
Nov 03 20:31:56.574622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:56.572577    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04160b7a-00d4-47d0-88dd-54646e94e9e5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4\" (UID: \"04160b7a-00d4-47d0-88dd-54646e94e9e5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4"
Nov 03 20:31:56.574622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:56.572657    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w7bg\" (UniqueName: \"kubernetes.io/projected/04160b7a-00d4-47d0-88dd-54646e94e9e5-kube-api-access-7w7bg\") pod \"payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4\" (UID: \"04160b7a-00d4-47d0-88dd-54646e94e9e5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4"
Nov 03 20:31:56.643428 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:56.643315    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w7bg\" (UniqueName: \"kubernetes.io/projected/04160b7a-00d4-47d0-88dd-54646e94e9e5-kube-api-access-7w7bg\") pod \"payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4\" (UID: \"04160b7a-00d4-47d0-88dd-54646e94e9e5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4"
Nov 03 20:31:56.724062 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:56.723143    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk" podUID="c9c3239a-d59c-4b81-95c6-b89eacc72111" containerName="payload-pre-processing" containerID="cri-o://d19e26ad6d5a3372c0dc278daee7a8b7c7d6ff78e5fb65026c1f65511787f85b" gracePeriod=25
Nov 03 20:31:56.835934 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:56.835905    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4"
Nov 03 20:31:56.877677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:56.876270    4421 patch_prober.go:29] interesting pod/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:31:56.877677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:56.876327    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf" podUID="32697d8f-a748-4deb-8e4c-5c8a08c235f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Nov 03 20:31:56.995365 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:56.995338    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8"
Nov 03 20:31:57.091668 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.089303    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcfb2deb-19e8-4407-90f9-d20452299a86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfb2deb-19e8-4407-90f9-d20452299a86-plugins-config-volume\") pod \"fcfb2deb-19e8-4407-90f9-d20452299a86\" (UID: \"fcfb2deb-19e8-4407-90f9-d20452299a86\") "
Nov 03 20:31:57.091668 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.089352    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcfb2deb-19e8-4407-90f9-d20452299a86-kube-api-access-mmgcb\" (UniqueName: \"kubernetes.io/projected/fcfb2deb-19e8-4407-90f9-d20452299a86-kube-api-access-mmgcb\") pod \"fcfb2deb-19e8-4407-90f9-d20452299a86\" (UID: \"fcfb2deb-19e8-4407-90f9-d20452299a86\") "
Nov 03 20:31:57.091918 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.091820    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcfb2deb-19e8-4407-90f9-d20452299a86-plugins-config-volume" pod "fcfb2deb-19e8-4407-90f9-d20452299a86" (UID: "fcfb2deb-19e8-4407-90f9-d20452299a86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:57.095748 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.095720    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcfb2deb-19e8-4407-90f9-d20452299a86-kube-api-access-mmgcb" pod "fcfb2deb-19e8-4407-90f9-d20452299a86" (UID: "fcfb2deb-19e8-4407-90f9-d20452299a86"). InnerVolumeSpecName "kube-api-access-mmgcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:57.192184 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.192154    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcfb2deb-19e8-4407-90f9-d20452299a86-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:57.192423 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.192403    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmgcb\" (UniqueName: \"kubernetes.io/projected/fcfb2deb-19e8-4407-90f9-d20452299a86-kube-api-access-mmgcb\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:57.297006 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.296741    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk"
Nov 03 20:31:57.394275 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.394238    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9c3239a-d59c-4b81-95c6-b89eacc72111-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c3239a-d59c-4b81-95c6-b89eacc72111-plugins-config-volume\") pod \"c9c3239a-d59c-4b81-95c6-b89eacc72111\" (UID: \"c9c3239a-d59c-4b81-95c6-b89eacc72111\") "
Nov 03 20:31:57.394575 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.394552    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c3239a-d59c-4b81-95c6-b89eacc72111-kube-api-access-hwsrm\" (UniqueName: \"kubernetes.io/projected/c9c3239a-d59c-4b81-95c6-b89eacc72111-kube-api-access-hwsrm\") pod \"c9c3239a-d59c-4b81-95c6-b89eacc72111\" (UID: \"c9c3239a-d59c-4b81-95c6-b89eacc72111\") "
Nov 03 20:31:57.395061 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.394711    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c3239a-d59c-4b81-95c6-b89eacc72111-plugins-config-volume" pod "c9c3239a-d59c-4b81-95c6-b89eacc72111" (UID: "c9c3239a-d59c-4b81-95c6-b89eacc72111"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:57.395182 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.395090    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9c3239a-d59c-4b81-95c6-b89eacc72111-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:57.397741 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.397711    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c3239a-d59c-4b81-95c6-b89eacc72111-kube-api-access-hwsrm" pod "c9c3239a-d59c-4b81-95c6-b89eacc72111" (UID: "c9c3239a-d59c-4b81-95c6-b89eacc72111"). InnerVolumeSpecName "kube-api-access-hwsrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:57.527742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.520943    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4"]
Nov 03 20:31:57.555742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.528971    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwsrm\" (UniqueName: \"kubernetes.io/projected/c9c3239a-d59c-4b81-95c6-b89eacc72111-kube-api-access-hwsrm\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:57.565580 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:31:57.565533    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04160b7a_00d4_47d0_88dd_54646e94e9e5.slice/crio-9269017a99b29e6d3ea0e35c347202d5a0903c38ba9e6eafdad71f86cda2c7e5 WatchSource:0}: Error finding container 9269017a99b29e6d3ea0e35c347202d5a0903c38ba9e6eafdad71f86cda2c7e5: Status 404 returned error can't find the container with id 9269017a99b29e6d3ea0e35c347202d5a0903c38ba9e6eafdad71f86cda2c7e5
Nov 03 20:31:57.779398 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.779261    4421 generic.go:356] "Generic (PLEG): container finished" podID="fcfb2deb-19e8-4407-90f9-d20452299a86" containerID="c4a98f9811a52e1181f604d28ead48a0b07858f624dff2e83b2fdc4408c8d625" exitCode=0
Nov 03 20:31:57.779398 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.779334    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8"
Nov 03 20:31:57.779398 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.779357    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8" event={"ID":"fcfb2deb-19e8-4407-90f9-d20452299a86","Type":"ContainerDied","Data":"c4a98f9811a52e1181f604d28ead48a0b07858f624dff2e83b2fdc4408c8d625"}
Nov 03 20:31:57.779812 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.779416    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8" event={"ID":"fcfb2deb-19e8-4407-90f9-d20452299a86","Type":"ContainerDied","Data":"c2c0cc8992d7d603b713e04fc366e4f6105fa975d18b305a3426782087efe6a3"}
Nov 03 20:31:57.779812 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.779442    4421 scope.go:122] "RemoveContainer" containerID="c4a98f9811a52e1181f604d28ead48a0b07858f624dff2e83b2fdc4408c8d625"
Nov 03 20:31:57.786591 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.785846    4421 generic.go:356] "Generic (PLEG): container finished" podID="c9c3239a-d59c-4b81-95c6-b89eacc72111" containerID="d19e26ad6d5a3372c0dc278daee7a8b7c7d6ff78e5fb65026c1f65511787f85b" exitCode=0
Nov 03 20:31:57.786591 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.785895    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk"
Nov 03 20:31:57.786591 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.785929    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk" event={"ID":"c9c3239a-d59c-4b81-95c6-b89eacc72111","Type":"ContainerDied","Data":"d19e26ad6d5a3372c0dc278daee7a8b7c7d6ff78e5fb65026c1f65511787f85b"}
Nov 03 20:31:57.786591 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.785987    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk" event={"ID":"c9c3239a-d59c-4b81-95c6-b89eacc72111","Type":"ContainerDied","Data":"c70bfd415078a9992933d906212d6ba7f988bacce6eb9fa5c3bba6889684f5c8"}
Nov 03 20:31:57.787673 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.787627    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4" event={"ID":"04160b7a-00d4-47d0-88dd-54646e94e9e5","Type":"ContainerStarted","Data":"e4d4f1e7533e64f8d82e462e21a8b8199cc268a71aba7cd2018e231eefc42ce5"}
Nov 03 20:31:57.787757 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.787673    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4" event={"ID":"04160b7a-00d4-47d0-88dd-54646e94e9e5","Type":"ContainerStarted","Data":"9269017a99b29e6d3ea0e35c347202d5a0903c38ba9e6eafdad71f86cda2c7e5"}
Nov 03 20:31:57.787757 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.787702    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4"
Nov 03 20:31:57.795796 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.795779    4421 scope.go:122] "RemoveContainer" containerID="c4a98f9811a52e1181f604d28ead48a0b07858f624dff2e83b2fdc4408c8d625"
Nov 03 20:31:57.796122 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:31:57.796100    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a98f9811a52e1181f604d28ead48a0b07858f624dff2e83b2fdc4408c8d625\": container with ID starting with c4a98f9811a52e1181f604d28ead48a0b07858f624dff2e83b2fdc4408c8d625 not found: ID does not exist" containerID="c4a98f9811a52e1181f604d28ead48a0b07858f624dff2e83b2fdc4408c8d625"
Nov 03 20:31:57.796209 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.796131    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a98f9811a52e1181f604d28ead48a0b07858f624dff2e83b2fdc4408c8d625"} err="failed to get container status \"c4a98f9811a52e1181f604d28ead48a0b07858f624dff2e83b2fdc4408c8d625\": rpc error: code = NotFound desc = could not find container \"c4a98f9811a52e1181f604d28ead48a0b07858f624dff2e83b2fdc4408c8d625\": container with ID starting with c4a98f9811a52e1181f604d28ead48a0b07858f624dff2e83b2fdc4408c8d625 not found: ID does not exist"
Nov 03 20:31:57.796209 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.796152    4421 scope.go:122] "RemoveContainer" containerID="d19e26ad6d5a3372c0dc278daee7a8b7c7d6ff78e5fb65026c1f65511787f85b"
Nov 03 20:31:57.812870 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.812759    4421 scope.go:122] "RemoveContainer" containerID="d19e26ad6d5a3372c0dc278daee7a8b7c7d6ff78e5fb65026c1f65511787f85b"
Nov 03 20:31:57.813150 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:31:57.813125    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d19e26ad6d5a3372c0dc278daee7a8b7c7d6ff78e5fb65026c1f65511787f85b\": container with ID starting with d19e26ad6d5a3372c0dc278daee7a8b7c7d6ff78e5fb65026c1f65511787f85b not found: ID does not exist" containerID="d19e26ad6d5a3372c0dc278daee7a8b7c7d6ff78e5fb65026c1f65511787f85b"
Nov 03 20:31:57.813248 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.813158    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d19e26ad6d5a3372c0dc278daee7a8b7c7d6ff78e5fb65026c1f65511787f85b"} err="failed to get container status \"d19e26ad6d5a3372c0dc278daee7a8b7c7d6ff78e5fb65026c1f65511787f85b\": rpc error: code = NotFound desc = could not find container \"d19e26ad6d5a3372c0dc278daee7a8b7c7d6ff78e5fb65026c1f65511787f85b\": container with ID starting with d19e26ad6d5a3372c0dc278daee7a8b7c7d6ff78e5fb65026c1f65511787f85b not found: ID does not exist"
Nov 03 20:31:57.837416 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.837371    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4" podStartSLOduration=1.837358058 podStartE2EDuration="1.837358058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:57.81281585 +0000 UTC m=+1847.945944312" watchObservedRunningTime="2026-11-03 20:31:57.837358058 +0000 UTC m=+1847.970486569"
Nov 03 20:31:57.837555 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.837524    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8"]
Nov 03 20:31:57.842125 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.842104    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-6bd876-5464b6d8d-ds4b8"]
Nov 03 20:31:57.852854 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.852831    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk"]
Nov 03 20:31:57.857301 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.857283    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-6bd876-c87b98655-9mjnk"]
Nov 03 20:31:57.858848 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.858825    4421 patch_prober.go:29] interesting pod/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Nov 03 20:31:57.858945 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:57.858878    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf" podUID="32697d8f-a748-4deb-8e4c-5c8a08c235f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Nov 03 20:31:58.075676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:58.075572    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9c3239a-d59c-4b81-95c6-b89eacc72111" path="/var/lib/kubelet/pods/c9c3239a-d59c-4b81-95c6-b89eacc72111/volumes"
Nov 03 20:31:58.076287 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:58.076263    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcfb2deb-19e8-4407-90f9-d20452299a86" path="/var/lib/kubelet/pods/fcfb2deb-19e8-4407-90f9-d20452299a86/volumes"
Nov 03 20:31:58.871129 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:58.871093    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:58.871532 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:58.871511    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:31:58.872324 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:31:58.872299    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 20:32:03.593620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.592644    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"]
Nov 03 20:32:03.641682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.624376    4421 cpu_manager.go:414] "containerMap: removing container" podUID="fcfb2deb-19e8-4407-90f9-d20452299a86" containerName="payload-processing"
Nov 03 20:32:03.641682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.624414    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcfb2deb-19e8-4407-90f9-d20452299a86" containerName="payload-processing"
Nov 03 20:32:03.641682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.624463    4421 cpu_manager.go:414] "containerMap: removing container" podUID="c9c3239a-d59c-4b81-95c6-b89eacc72111" containerName="payload-pre-processing"
Nov 03 20:32:03.641682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.624472    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c3239a-d59c-4b81-95c6-b89eacc72111" containerName="payload-pre-processing"
Nov 03 20:32:03.641682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.624714    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcfb2deb-19e8-4407-90f9-d20452299a86" containerName="payload-processing"
Nov 03 20:32:03.641682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.624734    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c3239a-d59c-4b81-95c6-b89eacc72111" containerName="payload-pre-processing"
Nov 03 20:32:03.699620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.688881    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"]
Nov 03 20:32:03.699620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.689027    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:03.710646 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.708798    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-9fbc61-openshift-default-dockercfg-v2stb\""
Nov 03 20:32:03.837755 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.837717    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fbddfde-9453-458d-b50b-05a3deeabf74-istiod-ca-cert\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-65cfr\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:03.840628 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.840577    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-envoy\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-65cfr\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:03.840891 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.840875    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-podinfo\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-65cfr\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:03.841000 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.840988    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-data\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-65cfr\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:03.841116 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.841102    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-credential-socket\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-65cfr\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:03.841221 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.841209    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9d6m\" (UniqueName: \"kubernetes.io/projected/4fbddfde-9453-458d-b50b-05a3deeabf74-kube-api-access-g9d6m\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-65cfr\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:03.841333 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.841320    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-token\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-65cfr\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:03.841573 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.841559    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-workload-certs\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-65cfr\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:03.841787 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.841768    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-workload-socket\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-65cfr\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:03.943934 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.943898    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fbddfde-9453-458d-b50b-05a3deeabf74-istiod-ca-cert\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-65cfr\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:03.944105 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.943947    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-envoy\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-65cfr\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:03.944105 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.944024    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-podinfo\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-65cfr\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:03.944105 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.944051    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-data\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-65cfr\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:03.944105 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.944086    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-credential-socket\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-65cfr\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:03.944332 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.944119    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9d6m\" (UniqueName: \"kubernetes.io/projected/4fbddfde-9453-458d-b50b-05a3deeabf74-kube-api-access-g9d6m\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-65cfr\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:03.944726 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.944697    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-data\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-65cfr\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:03.944860 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.944791    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fbddfde-9453-458d-b50b-05a3deeabf74-istiod-ca-cert\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-65cfr\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:03.945040 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.944969    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-credential-socket\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-65cfr\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:03.945040 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.945010    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-token\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-65cfr\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:03.945355 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.945145    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-workload-certs\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-65cfr\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:03.945355 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.945227    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-workload-socket\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-65cfr\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:03.945734 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.945710    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-workload-socket\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-65cfr\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:03.945734 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.945725    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-workload-certs\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-65cfr\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:03.947165 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.947142    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-podinfo\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-65cfr\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:03.947567 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.947544    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-envoy\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-65cfr\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:03.956519 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.956491    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9d6m\" (UniqueName: \"kubernetes.io/projected/4fbddfde-9453-458d-b50b-05a3deeabf74-kube-api-access-g9d6m\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-65cfr\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:03.959198 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:03.959172    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-token\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-65cfr\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:04.161275 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:04.161183    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:04.477905 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:04.477873    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"]
Nov 03 20:32:04.478225 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:32:04.478199    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fbddfde_9453_458d_b50b_05a3deeabf74.slice/crio-ec92531b3473cf3bc6eb1d590e1785eccce8049ee508cae31df5a85abc7da470 WatchSource:0}: Error finding container ec92531b3473cf3bc6eb1d590e1785eccce8049ee508cae31df5a85abc7da470: Status 404 returned error can't find the container with id ec92531b3473cf3bc6eb1d590e1785eccce8049ee508cae31df5a85abc7da470
Nov 03 20:32:04.481220 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:04.481185    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:32:04.481297 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:04.481251    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:32:04.481297 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:04.481280    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:32:05.192284 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:05.192237    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr" event={"ID":"4fbddfde-9453-458d-b50b-05a3deeabf74","Type":"ContainerStarted","Data":"805470c43d8e2f5741e50fc54f9579d00ba1df5ffcefb0db8d3d68d1197816ef"}
Nov 03 20:32:05.192755 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:05.192320    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr" event={"ID":"4fbddfde-9453-458d-b50b-05a3deeabf74","Type":"ContainerStarted","Data":"ec92531b3473cf3bc6eb1d590e1785eccce8049ee508cae31df5a85abc7da470"}
Nov 03 20:32:05.223723 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:05.223657    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr" podStartSLOduration=2.22364182 podStartE2EDuration="2.22364182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:05.219157285 +0000 UTC m=+1855.352285774" watchObservedRunningTime="2026-11-03 20:32:05.22364182 +0000 UTC m=+1855.356770282"
Nov 03 20:32:06.161919 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:06.161886    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:06.167618 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:06.167571    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:06.202494 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:06.202470    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:06.203398 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:06.203376    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:32:07.055078 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:07.055045    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4"]
Nov 03 20:32:07.059974 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:07.059955    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4"
Nov 03 20:32:07.069044 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:07.067788    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-0f6e09-dockercfg-82vsx\""
Nov 03 20:32:07.069044 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:07.067832    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-0f6e09\""
Nov 03 20:32:07.074431 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:07.074407    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4"]
Nov 03 20:32:07.084700 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:07.084650    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q"]
Nov 03 20:32:07.091490 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:07.091444    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q"
Nov 03 20:32:07.096696 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:07.096665    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q"]
Nov 03 20:32:07.147251 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:07.131932    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a020faaa-cf90-466e-8715-ccf5f987b158-plugins-config-volume\") pod \"payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q\" (UID: \"a020faaa-cf90-466e-8715-ccf5f987b158\") " pod="openshift-ingress/payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q"
Nov 03 20:32:07.147251 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:07.132157    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txtvw\" (UniqueName: \"kubernetes.io/projected/68f60111-d601-492f-b82e-9db18a818e3a-kube-api-access-txtvw\") pod \"payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4\" (UID: \"68f60111-d601-492f-b82e-9db18a818e3a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4"
Nov 03 20:32:07.147251 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:07.132284    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvxbh\" (UniqueName: \"kubernetes.io/projected/a020faaa-cf90-466e-8715-ccf5f987b158-kube-api-access-vvxbh\") pod \"payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q\" (UID: \"a020faaa-cf90-466e-8715-ccf5f987b158\") " pod="openshift-ingress/payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q"
Nov 03 20:32:07.147251 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:07.132484    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f60111-d601-492f-b82e-9db18a818e3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4\" (UID: \"68f60111-d601-492f-b82e-9db18a818e3a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4"
Nov 03 20:32:07.289641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:07.288508    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f60111-d601-492f-b82e-9db18a818e3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4\" (UID: \"68f60111-d601-492f-b82e-9db18a818e3a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4"
Nov 03 20:32:07.289641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:07.288567    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a020faaa-cf90-466e-8715-ccf5f987b158-plugins-config-volume\") pod \"payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q\" (UID: \"a020faaa-cf90-466e-8715-ccf5f987b158\") " pod="openshift-ingress/payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q"
Nov 03 20:32:07.289641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:07.288823    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txtvw\" (UniqueName: \"kubernetes.io/projected/68f60111-d601-492f-b82e-9db18a818e3a-kube-api-access-txtvw\") pod \"payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4\" (UID: \"68f60111-d601-492f-b82e-9db18a818e3a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4"
Nov 03 20:32:07.289641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:07.288964    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvxbh\" (UniqueName: \"kubernetes.io/projected/a020faaa-cf90-466e-8715-ccf5f987b158-kube-api-access-vvxbh\") pod \"payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q\" (UID: \"a020faaa-cf90-466e-8715-ccf5f987b158\") " pod="openshift-ingress/payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q"
Nov 03 20:32:07.296651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:07.290979    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f60111-d601-492f-b82e-9db18a818e3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4\" (UID: \"68f60111-d601-492f-b82e-9db18a818e3a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4"
Nov 03 20:32:07.296651 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:07.291773    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a020faaa-cf90-466e-8715-ccf5f987b158-plugins-config-volume\") pod \"payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q\" (UID: \"a020faaa-cf90-466e-8715-ccf5f987b158\") " pod="openshift-ingress/payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q"
Nov 03 20:32:07.322940 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:07.322293    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txtvw\" (UniqueName: \"kubernetes.io/projected/68f60111-d601-492f-b82e-9db18a818e3a-kube-api-access-txtvw\") pod \"payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4\" (UID: \"68f60111-d601-492f-b82e-9db18a818e3a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4"
Nov 03 20:32:07.331657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:07.326342    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvxbh\" (UniqueName: \"kubernetes.io/projected/a020faaa-cf90-466e-8715-ccf5f987b158-kube-api-access-vvxbh\") pod \"payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q\" (UID: \"a020faaa-cf90-466e-8715-ccf5f987b158\") " pod="openshift-ingress/payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q"
Nov 03 20:32:07.386636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:07.383223    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4"
Nov 03 20:32:07.417520 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:07.417473    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q"
Nov 03 20:32:07.810769 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:32:07.810728    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda020faaa_cf90_466e_8715_ccf5f987b158.slice/crio-3675d58ddba6f33bf9fca466c289f04f47acdc6b15675a19893c1bef3e7a8489 WatchSource:0}: Error finding container 3675d58ddba6f33bf9fca466c289f04f47acdc6b15675a19893c1bef3e7a8489: Status 404 returned error can't find the container with id 3675d58ddba6f33bf9fca466c289f04f47acdc6b15675a19893c1bef3e7a8489
Nov 03 20:32:07.811899 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:07.811876    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q"]
Nov 03 20:32:07.859154 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:07.859124    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4"]
Nov 03 20:32:07.859611 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:32:07.859569    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f60111_d601_492f_b82e_9db18a818e3a.slice/crio-d18c863db16d3284a09b5a144708fbeb9c4636a7f3e584d3bb921b833aba04d6 WatchSource:0}: Error finding container d18c863db16d3284a09b5a144708fbeb9c4636a7f3e584d3bb921b833aba04d6: Status 404 returned error can't find the container with id d18c863db16d3284a09b5a144708fbeb9c4636a7f3e584d3bb921b833aba04d6
Nov 03 20:32:08.332429 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:08.332387    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4" event={"ID":"68f60111-d601-492f-b82e-9db18a818e3a","Type":"ContainerStarted","Data":"146cdf7f9f9170c6e14ed967ff12200cb645f7df4610a3ee4e12bed3b3ccc805"}
Nov 03 20:32:08.333045 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:08.333015    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4"
Nov 03 20:32:08.333163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:08.333053    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4" event={"ID":"68f60111-d601-492f-b82e-9db18a818e3a","Type":"ContainerStarted","Data":"d18c863db16d3284a09b5a144708fbeb9c4636a7f3e584d3bb921b833aba04d6"}
Nov 03 20:32:08.336168 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:08.336135    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q" event={"ID":"a020faaa-cf90-466e-8715-ccf5f987b158","Type":"ContainerStarted","Data":"41a1a73f2362f10f45d0daa2b866817e7899dfcc121b27fd25f94cd8fcc4dcc3"}
Nov 03 20:32:08.336270 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:08.336196    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q" event={"ID":"a020faaa-cf90-466e-8715-ccf5f987b158","Type":"ContainerStarted","Data":"3675d58ddba6f33bf9fca466c289f04f47acdc6b15675a19893c1bef3e7a8489"}
Nov 03 20:32:08.336270 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:08.336228    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q"
Nov 03 20:32:08.352443 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:08.352403    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4" podStartSLOduration=1.352390369 podStartE2EDuration="1.352390369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:08.351786398 +0000 UTC m=+1858.484914861" watchObservedRunningTime="2026-11-03 20:32:08.352390369 +0000 UTC m=+1858.485518870"
Nov 03 20:32:08.372399 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:08.372361    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q" podStartSLOduration=1.372347815 podStartE2EDuration="1.372347815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:08.369476786 +0000 UTC m=+1858.502605244" watchObservedRunningTime="2026-11-03 20:32:08.372347815 +0000 UTC m=+1858.505476278"
Nov 03 20:32:08.839568 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:08.839541    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4"
Nov 03 20:32:09.031613 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:09.031560    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9"]
Nov 03 20:32:09.036556 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:09.036533    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg"]
Nov 03 20:32:09.036711 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:09.036682    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9"
Nov 03 20:32:09.044034 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:09.043043    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-9fbc61\""
Nov 03 20:32:09.044034 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:09.043154    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-9fbc61-dockercfg-4zb44\""
Nov 03 20:32:09.046857 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:09.046814    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z22k6\" (UniqueName: \"kubernetes.io/projected/f91c8aeb-d184-4b0c-bd49-7297d51cf271-kube-api-access-z22k6\") pod \"payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9\" (UID: \"f91c8aeb-d184-4b0c-bd49-7297d51cf271\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9"
Nov 03 20:32:09.047058 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:09.047038    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f91c8aeb-d184-4b0c-bd49-7297d51cf271-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9\" (UID: \"f91c8aeb-d184-4b0c-bd49-7297d51cf271\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9"
Nov 03 20:32:09.047563 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:09.047542    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9"]
Nov 03 20:32:09.047734 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:09.047687    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg"
Nov 03 20:32:09.056787 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:09.056755    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg"]
Nov 03 20:32:09.190770 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:09.167770    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z22k6\" (UniqueName: \"kubernetes.io/projected/f91c8aeb-d184-4b0c-bd49-7297d51cf271-kube-api-access-z22k6\") pod \"payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9\" (UID: \"f91c8aeb-d184-4b0c-bd49-7297d51cf271\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9"
Nov 03 20:32:09.196729 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:09.196698    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f91c8aeb-d184-4b0c-bd49-7297d51cf271-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9\" (UID: \"f91c8aeb-d184-4b0c-bd49-7297d51cf271\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9"
Nov 03 20:32:09.202784 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:09.202758    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6202aad3-67d4-4cec-8cbc-cdede3f9bb7c-plugins-config-volume\") pod \"payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg\" (UID: \"6202aad3-67d4-4cec-8cbc-cdede3f9bb7c\") " pod="openshift-ingress/payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg"
Nov 03 20:32:09.203070 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:09.203053    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6876g\" (UniqueName: \"kubernetes.io/projected/6202aad3-67d4-4cec-8cbc-cdede3f9bb7c-kube-api-access-6876g\") pod \"payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg\" (UID: \"6202aad3-67d4-4cec-8cbc-cdede3f9bb7c\") " pod="openshift-ingress/payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg"
Nov 03 20:32:09.203917 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:09.197646    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f91c8aeb-d184-4b0c-bd49-7297d51cf271-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9\" (UID: \"f91c8aeb-d184-4b0c-bd49-7297d51cf271\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9"
Nov 03 20:32:09.368987 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:09.307093    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6202aad3-67d4-4cec-8cbc-cdede3f9bb7c-plugins-config-volume\") pod \"payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg\" (UID: \"6202aad3-67d4-4cec-8cbc-cdede3f9bb7c\") " pod="openshift-ingress/payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg"
Nov 03 20:32:09.369737 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:09.369702    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6876g\" (UniqueName: \"kubernetes.io/projected/6202aad3-67d4-4cec-8cbc-cdede3f9bb7c-kube-api-access-6876g\") pod \"payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg\" (UID: \"6202aad3-67d4-4cec-8cbc-cdede3f9bb7c\") " pod="openshift-ingress/payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg"
Nov 03 20:32:09.370657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:09.308017    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6202aad3-67d4-4cec-8cbc-cdede3f9bb7c-plugins-config-volume\") pod \"payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg\" (UID: \"6202aad3-67d4-4cec-8cbc-cdede3f9bb7c\") " pod="openshift-ingress/payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg"
Nov 03 20:32:09.520562 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:09.520473    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6876g\" (UniqueName: \"kubernetes.io/projected/6202aad3-67d4-4cec-8cbc-cdede3f9bb7c-kube-api-access-6876g\") pod \"payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg\" (UID: \"6202aad3-67d4-4cec-8cbc-cdede3f9bb7c\") " pod="openshift-ingress/payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg"
Nov 03 20:32:09.537811 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:09.537738    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z22k6\" (UniqueName: \"kubernetes.io/projected/f91c8aeb-d184-4b0c-bd49-7297d51cf271-kube-api-access-z22k6\") pod \"payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9\" (UID: \"f91c8aeb-d184-4b0c-bd49-7297d51cf271\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9"
Nov 03 20:32:09.671665 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:09.671632    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9"
Nov 03 20:32:09.813983 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:09.813899    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg"
Nov 03 20:32:09.978276 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:32:09.978231    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf91c8aeb_d184_4b0c_bd49_7297d51cf271.slice/crio-3f8eaf63910b47984fe405bab3fae751cee0e9c369c380c81480029be21d2e04 WatchSource:0}: Error finding container 3f8eaf63910b47984fe405bab3fae751cee0e9c369c380c81480029be21d2e04: Status 404 returned error can't find the container with id 3f8eaf63910b47984fe405bab3fae751cee0e9c369c380c81480029be21d2e04
Nov 03 20:32:09.978713 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:09.978678    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9"]
Nov 03 20:32:10.105190 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:10.105162    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg"]
Nov 03 20:32:10.118348 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:32:10.118192    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6202aad3_67d4_4cec_8cbc_cdede3f9bb7c.slice/crio-6fd35673098bab0273c3fed960d048c9393fd5f906d0365695f1243a28602b31 WatchSource:0}: Error finding container 6fd35673098bab0273c3fed960d048c9393fd5f906d0365695f1243a28602b31: Status 404 returned error can't find the container with id 6fd35673098bab0273c3fed960d048c9393fd5f906d0365695f1243a28602b31
Nov 03 20:32:10.621449 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:10.621302    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg" event={"ID":"6202aad3-67d4-4cec-8cbc-cdede3f9bb7c","Type":"ContainerStarted","Data":"b03e0696a9d1536263b4dd757936cb855706273e952a478b498bcbf8722974d3"}
Nov 03 20:32:10.621919 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:10.621489    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg"
Nov 03 20:32:10.621919 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:10.621521    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg" event={"ID":"6202aad3-67d4-4cec-8cbc-cdede3f9bb7c","Type":"ContainerStarted","Data":"6fd35673098bab0273c3fed960d048c9393fd5f906d0365695f1243a28602b31"}
Nov 03 20:32:10.624749 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:10.624714    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9" event={"ID":"f91c8aeb-d184-4b0c-bd49-7297d51cf271","Type":"ContainerStarted","Data":"910c741a7b67da55e14e1c32fd9afa0bb68aa9030872c5f189ce92e17f531316"}
Nov 03 20:32:10.624889 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:10.624773    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9" event={"ID":"f91c8aeb-d184-4b0c-bd49-7297d51cf271","Type":"ContainerStarted","Data":"3f8eaf63910b47984fe405bab3fae751cee0e9c369c380c81480029be21d2e04"}
Nov 03 20:32:10.624889 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:10.624819    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9"
Nov 03 20:32:10.651897 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:10.651843    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg" podStartSLOduration=1.6518247339999999 podStartE2EDuration="1.651824734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:10.644155176 +0000 UTC m=+1860.777283641" watchObservedRunningTime="2026-11-03 20:32:10.651824734 +0000 UTC m=+1860.784953197"
Nov 03 20:32:10.670693 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:10.670647    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9" podStartSLOduration=1.6706316879999998 podStartE2EDuration="1.670631688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:10.665738195 +0000 UTC m=+1860.798866658" watchObservedRunningTime="2026-11-03 20:32:10.670631688 +0000 UTC m=+1860.803760152"
Nov 03 20:32:16.296454 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:16.296396    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"]
Nov 03 20:32:16.296981 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:16.296703    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87" podUID="57010672-22b7-4c7e-a3c0-75578515deb9" containerName="istio-proxy" containerID="cri-o://443a3fcaeafe3d2e714a3b8165f476a77e184f3c9ad4d2ce7a43c1aa8eff8c3d" gracePeriod=30
Nov 03 20:32:16.306090 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:16.306064    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"]
Nov 03 20:32:16.306328 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:16.306294    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8" podUID="39b8d333-b40a-4096-9de9-a944842f9544" containerName="istio-proxy" containerID="cri-o://8bb74a324f58b9f916e189f4656732c4ee7ba2a2fe2ed6c4522e459c15123f19" gracePeriod=30
Nov 03 20:32:19.519676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:19.519650    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q"
Nov 03 20:32:19.631331 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:32:19.631296    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:32:20.677621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:20.670827    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4"
Nov 03 20:32:21.619773 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.619692    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:32:21.636525 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.636498    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg"
Nov 03 20:32:21.636717 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.636619    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9"
Nov 03 20:32:21.760059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.751302    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57010672-22b7-4c7e-a3c0-75578515deb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57010672-22b7-4c7e-a3c0-75578515deb9-istio-podinfo\") pod \"57010672-22b7-4c7e-a3c0-75578515deb9\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") "
Nov 03 20:32:21.760059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.751520    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57010672-22b7-4c7e-a3c0-75578515deb9-istio-token\" (UniqueName: \"kubernetes.io/projected/57010672-22b7-4c7e-a3c0-75578515deb9-istio-token\") pod \"57010672-22b7-4c7e-a3c0-75578515deb9\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") "
Nov 03 20:32:21.760059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.751589    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57010672-22b7-4c7e-a3c0-75578515deb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57010672-22b7-4c7e-a3c0-75578515deb9-istiod-ca-cert\") pod \"57010672-22b7-4c7e-a3c0-75578515deb9\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") "
Nov 03 20:32:21.760059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.751689    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-credential-socket\") pod \"57010672-22b7-4c7e-a3c0-75578515deb9\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") "
Nov 03 20:32:21.760059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.751859    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57010672-22b7-4c7e-a3c0-75578515deb9-kube-api-access-75bkg\" (UniqueName: \"kubernetes.io/projected/57010672-22b7-4c7e-a3c0-75578515deb9-kube-api-access-75bkg\") pod \"57010672-22b7-4c7e-a3c0-75578515deb9\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") "
Nov 03 20:32:21.760059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.751888    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-workload-certs\") pod \"57010672-22b7-4c7e-a3c0-75578515deb9\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") "
Nov 03 20:32:21.760059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.751918    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-istio-data\") pod \"57010672-22b7-4c7e-a3c0-75578515deb9\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") "
Nov 03 20:32:21.760059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.751947    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-istio-envoy\") pod \"57010672-22b7-4c7e-a3c0-75578515deb9\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") "
Nov 03 20:32:21.760059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.751997    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-workload-socket\") pod \"57010672-22b7-4c7e-a3c0-75578515deb9\" (UID: \"57010672-22b7-4c7e-a3c0-75578515deb9\") "
Nov 03 20:32:21.760059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.754836    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-workload-socket" pod "57010672-22b7-4c7e-a3c0-75578515deb9" (UID: "57010672-22b7-4c7e-a3c0-75578515deb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:21.760059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.754982    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-workload-certs" pod "57010672-22b7-4c7e-a3c0-75578515deb9" (UID: "57010672-22b7-4c7e-a3c0-75578515deb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:21.760059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.755106    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-istio-data" pod "57010672-22b7-4c7e-a3c0-75578515deb9" (UID: "57010672-22b7-4c7e-a3c0-75578515deb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:21.760059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.756680    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-credential-socket" pod "57010672-22b7-4c7e-a3c0-75578515deb9" (UID: "57010672-22b7-4c7e-a3c0-75578515deb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:21.760059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.758944    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57010672-22b7-4c7e-a3c0-75578515deb9-istiod-ca-cert" pod "57010672-22b7-4c7e-a3c0-75578515deb9" (UID: "57010672-22b7-4c7e-a3c0-75578515deb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:21.812005 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.811052    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57010672-22b7-4c7e-a3c0-75578515deb9-istio-podinfo" pod "57010672-22b7-4c7e-a3c0-75578515deb9" (UID: "57010672-22b7-4c7e-a3c0-75578515deb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:32:21.812005 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.811377    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-istio-envoy" pod "57010672-22b7-4c7e-a3c0-75578515deb9" (UID: "57010672-22b7-4c7e-a3c0-75578515deb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:21.812005 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.811997    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57010672-22b7-4c7e-a3c0-75578515deb9-kube-api-access-75bkg" pod "57010672-22b7-4c7e-a3c0-75578515deb9" (UID: "57010672-22b7-4c7e-a3c0-75578515deb9"). InnerVolumeSpecName "kube-api-access-75bkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:21.812343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.812042    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57010672-22b7-4c7e-a3c0-75578515deb9-istio-token" pod "57010672-22b7-4c7e-a3c0-75578515deb9" (UID: "57010672-22b7-4c7e-a3c0-75578515deb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:21.850346 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.850318    4421 generic.go:356] "Generic (PLEG): container finished" podID="57010672-22b7-4c7e-a3c0-75578515deb9" containerID="443a3fcaeafe3d2e714a3b8165f476a77e184f3c9ad4d2ce7a43c1aa8eff8c3d" exitCode=0
Nov 03 20:32:21.850612 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.850576    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87" event={"ID":"57010672-22b7-4c7e-a3c0-75578515deb9","Type":"ContainerDied","Data":"443a3fcaeafe3d2e714a3b8165f476a77e184f3c9ad4d2ce7a43c1aa8eff8c3d"}
Nov 03 20:32:21.850744 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.850732    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87" event={"ID":"57010672-22b7-4c7e-a3c0-75578515deb9","Type":"ContainerDied","Data":"b4205acf6236434ac6e19178db2d5c88b76dce6514897f983bf61863fb5c9675"}
Nov 03 20:32:21.850827 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.850819    4421 scope.go:122] "RemoveContainer" containerID="443a3fcaeafe3d2e714a3b8165f476a77e184f3c9ad4d2ce7a43c1aa8eff8c3d"
Nov 03 20:32:21.851007 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.850996    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"
Nov 03 20:32:21.861142 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.859415    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57010672-22b7-4c7e-a3c0-75578515deb9-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:21.861142 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.859440    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57010672-22b7-4c7e-a3c0-75578515deb9-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:21.861142 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.859453    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57010672-22b7-4c7e-a3c0-75578515deb9-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:21.861142 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.859467    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:21.861142 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.859480    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75bkg\" (UniqueName: \"kubernetes.io/projected/57010672-22b7-4c7e-a3c0-75578515deb9-kube-api-access-75bkg\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:21.861142 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.859493    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:21.861142 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.859507    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:21.861142 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.859521    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:21.861142 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.859533    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57010672-22b7-4c7e-a3c0-75578515deb9-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:21.865909 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.865874    4421 generic.go:356] "Generic (PLEG): container finished" podID="39b8d333-b40a-4096-9de9-a944842f9544" containerID="8bb74a324f58b9f916e189f4656732c4ee7ba2a2fe2ed6c4522e459c15123f19" exitCode=0
Nov 03 20:32:21.866086 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.866065    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8" event={"ID":"39b8d333-b40a-4096-9de9-a944842f9544","Type":"ContainerDied","Data":"8bb74a324f58b9f916e189f4656732c4ee7ba2a2fe2ed6c4522e459c15123f19"}
Nov 03 20:32:21.887783 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.887747    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"]
Nov 03 20:32:21.889910 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.889888    4421 scope.go:122] "RemoveContainer" containerID="443a3fcaeafe3d2e714a3b8165f476a77e184f3c9ad4d2ce7a43c1aa8eff8c3d"
Nov 03 20:32:21.892914 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:32:21.892821    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"443a3fcaeafe3d2e714a3b8165f476a77e184f3c9ad4d2ce7a43c1aa8eff8c3d\": container with ID starting with 443a3fcaeafe3d2e714a3b8165f476a77e184f3c9ad4d2ce7a43c1aa8eff8c3d not found: ID does not exist" containerID="443a3fcaeafe3d2e714a3b8165f476a77e184f3c9ad4d2ce7a43c1aa8eff8c3d"
Nov 03 20:32:21.892914 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.892854    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"443a3fcaeafe3d2e714a3b8165f476a77e184f3c9ad4d2ce7a43c1aa8eff8c3d"} err="failed to get container status \"443a3fcaeafe3d2e714a3b8165f476a77e184f3c9ad4d2ce7a43c1aa8eff8c3d\": rpc error: code = NotFound desc = could not find container \"443a3fcaeafe3d2e714a3b8165f476a77e184f3c9ad4d2ce7a43c1aa8eff8c3d\": container with ID starting with 443a3fcaeafe3d2e714a3b8165f476a77e184f3c9ad4d2ce7a43c1aa8eff8c3d not found: ID does not exist"
Nov 03 20:32:21.893380 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:21.893336    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ghh87"]
Nov 03 20:32:22.027437 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.027390    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57010672-22b7-4c7e-a3c0-75578515deb9" path="/var/lib/kubelet/pods/57010672-22b7-4c7e-a3c0-75578515deb9/volumes"
Nov 03 20:32:22.066353 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.066331    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:32:22.163996 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.163873    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39b8d333-b40a-4096-9de9-a944842f9544-istio-token\" (UniqueName: \"kubernetes.io/projected/39b8d333-b40a-4096-9de9-a944842f9544-istio-token\") pod \"39b8d333-b40a-4096-9de9-a944842f9544\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") "
Nov 03 20:32:22.165982 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.164041    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-istio-data\") pod \"39b8d333-b40a-4096-9de9-a944842f9544\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") "
Nov 03 20:32:22.165982 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.164112    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39b8d333-b40a-4096-9de9-a944842f9544-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b8d333-b40a-4096-9de9-a944842f9544-istio-podinfo\") pod \"39b8d333-b40a-4096-9de9-a944842f9544\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") "
Nov 03 20:32:22.165982 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.164310    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-workload-certs\") pod \"39b8d333-b40a-4096-9de9-a944842f9544\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") "
Nov 03 20:32:22.165982 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.164358    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39b8d333-b40a-4096-9de9-a944842f9544-kube-api-access-t6cvq\" (UniqueName: \"kubernetes.io/projected/39b8d333-b40a-4096-9de9-a944842f9544-kube-api-access-t6cvq\") pod \"39b8d333-b40a-4096-9de9-a944842f9544\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") "
Nov 03 20:32:22.165982 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.164384    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-credential-socket\") pod \"39b8d333-b40a-4096-9de9-a944842f9544\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") "
Nov 03 20:32:22.165982 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.164508    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-workload-socket\") pod \"39b8d333-b40a-4096-9de9-a944842f9544\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") "
Nov 03 20:32:22.165982 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.164535    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39b8d333-b40a-4096-9de9-a944842f9544-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b8d333-b40a-4096-9de9-a944842f9544-istiod-ca-cert\") pod \"39b8d333-b40a-4096-9de9-a944842f9544\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") "
Nov 03 20:32:22.165982 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.164537    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-istio-data" pod "39b8d333-b40a-4096-9de9-a944842f9544" (UID: "39b8d333-b40a-4096-9de9-a944842f9544"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:22.165982 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.165665    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-workload-certs" pod "39b8d333-b40a-4096-9de9-a944842f9544" (UID: "39b8d333-b40a-4096-9de9-a944842f9544"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:22.165982 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.165745    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-workload-socket" pod "39b8d333-b40a-4096-9de9-a944842f9544" (UID: "39b8d333-b40a-4096-9de9-a944842f9544"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:22.165982 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.165967    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39b8d333-b40a-4096-9de9-a944842f9544-istiod-ca-cert" pod "39b8d333-b40a-4096-9de9-a944842f9544" (UID: "39b8d333-b40a-4096-9de9-a944842f9544"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:22.166817 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.166263    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-credential-socket" pod "39b8d333-b40a-4096-9de9-a944842f9544" (UID: "39b8d333-b40a-4096-9de9-a944842f9544"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:22.166817 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.166359    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-istio-envoy\") pod \"39b8d333-b40a-4096-9de9-a944842f9544\" (UID: \"39b8d333-b40a-4096-9de9-a944842f9544\") "
Nov 03 20:32:22.167612 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.167564    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b8d333-b40a-4096-9de9-a944842f9544-kube-api-access-t6cvq" pod "39b8d333-b40a-4096-9de9-a944842f9544" (UID: "39b8d333-b40a-4096-9de9-a944842f9544"). InnerVolumeSpecName "kube-api-access-t6cvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:22.168477 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.168033    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b8d333-b40a-4096-9de9-a944842f9544-istio-token" pod "39b8d333-b40a-4096-9de9-a944842f9544" (UID: "39b8d333-b40a-4096-9de9-a944842f9544"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:22.168477 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.168364    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:22.168477 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.168388    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6cvq\" (UniqueName: \"kubernetes.io/projected/39b8d333-b40a-4096-9de9-a944842f9544-kube-api-access-t6cvq\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:22.168477 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.168402    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:22.168477 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.168416    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:22.168477 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.168429    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b8d333-b40a-4096-9de9-a944842f9544-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:22.168477 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.168441    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39b8d333-b40a-4096-9de9-a944842f9544-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:22.168477 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.168454    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:22.169844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.169248    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-istio-envoy" pod "39b8d333-b40a-4096-9de9-a944842f9544" (UID: "39b8d333-b40a-4096-9de9-a944842f9544"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:22.170057 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.170031    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39b8d333-b40a-4096-9de9-a944842f9544-istio-podinfo" pod "39b8d333-b40a-4096-9de9-a944842f9544" (UID: "39b8d333-b40a-4096-9de9-a944842f9544"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:32:22.270756 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.270728    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b8d333-b40a-4096-9de9-a944842f9544-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:22.270756 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.270753    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b8d333-b40a-4096-9de9-a944842f9544-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:22.891748 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.891711    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8" event={"ID":"39b8d333-b40a-4096-9de9-a944842f9544","Type":"ContainerDied","Data":"af1c12e2c885a64d22a2383253c22edfbd385442447513574bf83475f44306b6"}
Nov 03 20:32:22.892303 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.892283    4421 scope.go:122] "RemoveContainer" containerID="8bb74a324f58b9f916e189f4656732c4ee7ba2a2fe2ed6c4522e459c15123f19"
Nov 03 20:32:22.892546 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.892532    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"
Nov 03 20:32:22.975444 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.975404    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"]
Nov 03 20:32:22.990656 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:22.982018    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wlhq8"]
Nov 03 20:32:23.946015 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:23.945984    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2"]
Nov 03 20:32:23.948591 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:23.948561    4421 cpu_manager.go:414] "containerMap: removing container" podUID="57010672-22b7-4c7e-a3c0-75578515deb9" containerName="istio-proxy"
Nov 03 20:32:23.948782 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:23.948765    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57010672-22b7-4c7e-a3c0-75578515deb9" containerName="istio-proxy"
Nov 03 20:32:23.948915 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:23.948903    4421 cpu_manager.go:414] "containerMap: removing container" podUID="39b8d333-b40a-4096-9de9-a944842f9544" containerName="istio-proxy"
Nov 03 20:32:23.949015 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:23.949002    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39b8d333-b40a-4096-9de9-a944842f9544" containerName="istio-proxy"
Nov 03 20:32:23.949418 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:23.949398    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="57010672-22b7-4c7e-a3c0-75578515deb9" containerName="istio-proxy"
Nov 03 20:32:23.949534 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:23.949520    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="39b8d333-b40a-4096-9de9-a944842f9544" containerName="istio-proxy"
Nov 03 20:32:23.988181 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:23.988152    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2"
Nov 03 20:32:24.039517 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:24.039489    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-9fbc61-serving-cert\""
Nov 03 20:32:24.065940 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:24.065906    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4524e977-ff54-4424-bcdb-28764428b1ef-maas-api-tls\") pod \"maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2\" (UID: \"4524e977-ff54-4424-bcdb-28764428b1ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2"
Nov 03 20:32:24.066488 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:24.066458    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrcsd\" (UniqueName: \"kubernetes.io/projected/4524e977-ff54-4424-bcdb-28764428b1ef-kube-api-access-jrcsd\") pod \"maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2\" (UID: \"4524e977-ff54-4424-bcdb-28764428b1ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2"
Nov 03 20:32:24.066704 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:24.066684    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4524e977-ff54-4424-bcdb-28764428b1ef-openshift-service-ca\") pod \"maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2\" (UID: \"4524e977-ff54-4424-bcdb-28764428b1ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2"
Nov 03 20:32:24.066994 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:24.066976    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4524e977-ff54-4424-bcdb-28764428b1ef-metrics-certs\") pod \"maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2\" (UID: \"4524e977-ff54-4424-bcdb-28764428b1ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2"
Nov 03 20:32:24.195625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:24.184262    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrcsd\" (UniqueName: \"kubernetes.io/projected/4524e977-ff54-4424-bcdb-28764428b1ef-kube-api-access-jrcsd\") pod \"maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2\" (UID: \"4524e977-ff54-4424-bcdb-28764428b1ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2"
Nov 03 20:32:24.195625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:24.184348    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4524e977-ff54-4424-bcdb-28764428b1ef-openshift-service-ca\") pod \"maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2\" (UID: \"4524e977-ff54-4424-bcdb-28764428b1ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2"
Nov 03 20:32:24.195625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:24.184583    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4524e977-ff54-4424-bcdb-28764428b1ef-metrics-certs\") pod \"maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2\" (UID: \"4524e977-ff54-4424-bcdb-28764428b1ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2"
Nov 03 20:32:24.195625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:24.184653    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4524e977-ff54-4424-bcdb-28764428b1ef-maas-api-tls\") pod \"maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2\" (UID: \"4524e977-ff54-4424-bcdb-28764428b1ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2"
Nov 03 20:32:24.195625 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:32:24.185146    4421 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-serving-cert: secret "maas-api-e2e-autores-9fbc61-serving-cert" not found
Nov 03 20:32:24.195625 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:32:24.185213    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4524e977-ff54-4424-bcdb-28764428b1ef-maas-api-tls podName:4524e977-ff54-4424-bcdb-28764428b1ef nodeName:}" failed. No retries permitted until 2026-11-03 20:32:24.685193803 +0000 UTC m=+1874.818322249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4524e977-ff54-4424-bcdb-28764428b1ef-maas-api-tls") pod "maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2" (UID: "4524e977-ff54-4424-bcdb-28764428b1ef") : secret "maas-api-e2e-autores-9fbc61-serving-cert" not found
Nov 03 20:32:24.195625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:24.186361    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4524e977-ff54-4424-bcdb-28764428b1ef-openshift-service-ca\") pod \"maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2\" (UID: \"4524e977-ff54-4424-bcdb-28764428b1ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2"
Nov 03 20:32:24.202623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:24.202378    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39b8d333-b40a-4096-9de9-a944842f9544" path="/var/lib/kubelet/pods/39b8d333-b40a-4096-9de9-a944842f9544/volumes"
Nov 03 20:32:24.217829 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:24.217800    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2"]
Nov 03 20:32:24.288623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:24.285269    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4524e977-ff54-4424-bcdb-28764428b1ef-metrics-certs\") pod \"maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2\" (UID: \"4524e977-ff54-4424-bcdb-28764428b1ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2"
Nov 03 20:32:24.352621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:24.347970    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrcsd\" (UniqueName: \"kubernetes.io/projected/4524e977-ff54-4424-bcdb-28764428b1ef-kube-api-access-jrcsd\") pod \"maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2\" (UID: \"4524e977-ff54-4424-bcdb-28764428b1ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2"
Nov 03 20:32:24.733621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:24.721680    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4524e977-ff54-4424-bcdb-28764428b1ef-maas-api-tls\") pod \"maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2\" (UID: \"4524e977-ff54-4424-bcdb-28764428b1ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2"
Nov 03 20:32:24.768173 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:24.764457    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4524e977-ff54-4424-bcdb-28764428b1ef-maas-api-tls\") pod \"maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2\" (UID: \"4524e977-ff54-4424-bcdb-28764428b1ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2"
Nov 03 20:32:24.818644 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:24.817355    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2"
Nov 03 20:32:25.314625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:25.302840    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2"]
Nov 03 20:32:25.340754 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:25.329981    4421 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:32:25.997363 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:25.997322    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2" event={"ID":"4524e977-ff54-4424-bcdb-28764428b1ef","Type":"ContainerStarted","Data":"d44beb1a938ffffbcaf9c9eea9b9b7e93ead4fba26724187233f30f429641886"}
Nov 03 20:32:25.997564 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:25.997389    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2" event={"ID":"4524e977-ff54-4424-bcdb-28764428b1ef","Type":"ContainerStarted","Data":"5f2eade1e0c99e2c904a8ff7b24d07ba3a5509ee6eed6ad0513c12bb6515942d"}
Nov 03 20:32:25.997564 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:25.997417    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2"
Nov 03 20:32:26.018411 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:26.018326    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2" podStartSLOduration=2.599807106 podStartE2EDuration="3.018312092s" totalImagesPullingTime="418.504986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:26.016560913 +0000 UTC m=+1876.149689376" watchObservedRunningTime="2026-11-03 20:32:26.018312092 +0000 UTC m=+1876.151440589"
Nov 03 20:32:32.079081 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:32.078900    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2"
Nov 03 20:32:36.609190 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:36.609150    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89"]
Nov 03 20:32:36.615906 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:36.615878    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89"
Nov 03 20:32:36.619916 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:36.619887    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-db5198-serving-cert\""
Nov 03 20:32:36.625096 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:36.625067    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89"]
Nov 03 20:32:36.691715 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:36.691678    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a288a11-b106-4ddc-a272-fe47967dacab-metrics-certs\") pod \"maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89\" (UID: \"4a288a11-b106-4ddc-a272-fe47967dacab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89"
Nov 03 20:32:36.691901 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:36.691738    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a288a11-b106-4ddc-a272-fe47967dacab-maas-api-tls\") pod \"maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89\" (UID: \"4a288a11-b106-4ddc-a272-fe47967dacab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89"
Nov 03 20:32:36.691969 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:36.691908    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a288a11-b106-4ddc-a272-fe47967dacab-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89\" (UID: \"4a288a11-b106-4ddc-a272-fe47967dacab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89"
Nov 03 20:32:36.692025 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:36.692008    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dggkr\" (UniqueName: \"kubernetes.io/projected/4a288a11-b106-4ddc-a272-fe47967dacab-kube-api-access-dggkr\") pod \"maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89\" (UID: \"4a288a11-b106-4ddc-a272-fe47967dacab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89"
Nov 03 20:32:36.795069 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:36.795006    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a288a11-b106-4ddc-a272-fe47967dacab-metrics-certs\") pod \"maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89\" (UID: \"4a288a11-b106-4ddc-a272-fe47967dacab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89"
Nov 03 20:32:36.795378 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:36.795350    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a288a11-b106-4ddc-a272-fe47967dacab-maas-api-tls\") pod \"maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89\" (UID: \"4a288a11-b106-4ddc-a272-fe47967dacab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89"
Nov 03 20:32:36.816161 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:36.816126    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a288a11-b106-4ddc-a272-fe47967dacab-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89\" (UID: \"4a288a11-b106-4ddc-a272-fe47967dacab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89"
Nov 03 20:32:36.816428 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:36.816410    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dggkr\" (UniqueName: \"kubernetes.io/projected/4a288a11-b106-4ddc-a272-fe47967dacab-kube-api-access-dggkr\") pod \"maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89\" (UID: \"4a288a11-b106-4ddc-a272-fe47967dacab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89"
Nov 03 20:32:36.829235 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:32:36.806794    4421 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-serving-cert: secret "maas-api-e2e-extmod-a-db5198-serving-cert" not found
Nov 03 20:32:36.829615 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:32:36.829576    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a288a11-b106-4ddc-a272-fe47967dacab-maas-api-tls podName:4a288a11-b106-4ddc-a272-fe47967dacab nodeName:}" failed. No retries permitted until 2026-11-03 20:32:37.329549778 +0000 UTC m=+1887.462678239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4a288a11-b106-4ddc-a272-fe47967dacab-maas-api-tls") pod "maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89" (UID: "4a288a11-b106-4ddc-a272-fe47967dacab") : secret "maas-api-e2e-extmod-a-db5198-serving-cert" not found
Nov 03 20:32:36.842064 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:36.830747    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a288a11-b106-4ddc-a272-fe47967dacab-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89\" (UID: \"4a288a11-b106-4ddc-a272-fe47967dacab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89"
Nov 03 20:32:36.865673 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:36.847564    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a288a11-b106-4ddc-a272-fe47967dacab-metrics-certs\") pod \"maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89\" (UID: \"4a288a11-b106-4ddc-a272-fe47967dacab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89"
Nov 03 20:32:36.895535 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:36.895309    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dggkr\" (UniqueName: \"kubernetes.io/projected/4a288a11-b106-4ddc-a272-fe47967dacab-kube-api-access-dggkr\") pod \"maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89\" (UID: \"4a288a11-b106-4ddc-a272-fe47967dacab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89"
Nov 03 20:32:37.329802 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:37.329764    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a288a11-b106-4ddc-a272-fe47967dacab-maas-api-tls\") pod \"maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89\" (UID: \"4a288a11-b106-4ddc-a272-fe47967dacab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89"
Nov 03 20:32:37.333028 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:37.332991    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a288a11-b106-4ddc-a272-fe47967dacab-maas-api-tls\") pod \"maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89\" (UID: \"4a288a11-b106-4ddc-a272-fe47967dacab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89"
Nov 03 20:32:37.659012 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:37.658919    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89"
Nov 03 20:32:38.094564 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:38.094536    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89"]
Nov 03 20:32:38.095130 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:32:38.095108    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a288a11_b106_4ddc_a272_fe47967dacab.slice/crio-1193c8de501610314574c5fc643357cfb58b6a33f6ab7c7cdf9c618cdea032c4 WatchSource:0}: Error finding container 1193c8de501610314574c5fc643357cfb58b6a33f6ab7c7cdf9c618cdea032c4: Status 404 returned error can't find the container with id 1193c8de501610314574c5fc643357cfb58b6a33f6ab7c7cdf9c618cdea032c4
Nov 03 20:32:38.397683 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:38.386042    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89" event={"ID":"4a288a11-b106-4ddc-a272-fe47967dacab","Type":"ContainerStarted","Data":"1193c8de501610314574c5fc643357cfb58b6a33f6ab7c7cdf9c618cdea032c4"}
Nov 03 20:32:39.397192 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:39.397148    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89" event={"ID":"4a288a11-b106-4ddc-a272-fe47967dacab","Type":"ContainerStarted","Data":"bc6adce0a5f952263895b0102217d948d00c2049ddd246cefffdc7231ddafde5"}
Nov 03 20:32:39.397679 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:39.397295    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89"
Nov 03 20:32:39.419419 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:39.419367    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89" podStartSLOduration=2.8828260439999998 podStartE2EDuration="3.419353405s" totalImagesPullingTime="536.527361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:39.416818634 +0000 UTC m=+1889.549947098" watchObservedRunningTime="2026-11-03 20:32:39.419353405 +0000 UTC m=+1889.552481866"
Nov 03 20:32:41.180028 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:41.180000    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"]
Nov 03 20:32:41.180585 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:41.180207    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9" podUID="7eefc668-8d72-4ee2-b047-91d13318ec49" containerName="istio-proxy" containerID="cri-o://b01881ec6dfe12f1dbc37fff50f48ad859e9531b5e004f87d0c831acae2a64dd" gracePeriod=30
Nov 03 20:32:41.186068 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:41.186039    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"]
Nov 03 20:32:41.186234 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:41.186212    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7" podUID="61f598b0-63c5-449a-b963-3d9921553aae" containerName="istio-proxy" containerID="cri-o://d017ce5afee266d337a066c74c14643a2fedfdeac2a3d5b1b85406aed27511fb" gracePeriod=30
Nov 03 20:32:45.410991 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:45.410958    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89"
Nov 03 20:32:46.546900 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.546859    4421 generic.go:356] "Generic (PLEG): container finished" podID="61f598b0-63c5-449a-b963-3d9921553aae" containerID="d017ce5afee266d337a066c74c14643a2fedfdeac2a3d5b1b85406aed27511fb" exitCode=0
Nov 03 20:32:46.547374 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.546936    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7" event={"ID":"61f598b0-63c5-449a-b963-3d9921553aae","Type":"ContainerDied","Data":"d017ce5afee266d337a066c74c14643a2fedfdeac2a3d5b1b85406aed27511fb"}
Nov 03 20:32:46.548476 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.548433    4421 generic.go:356] "Generic (PLEG): container finished" podID="7eefc668-8d72-4ee2-b047-91d13318ec49" containerID="b01881ec6dfe12f1dbc37fff50f48ad859e9531b5e004f87d0c831acae2a64dd" exitCode=0
Nov 03 20:32:46.548653 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.548489    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9" event={"ID":"7eefc668-8d72-4ee2-b047-91d13318ec49","Type":"ContainerDied","Data":"b01881ec6dfe12f1dbc37fff50f48ad859e9531b5e004f87d0c831acae2a64dd"}
Nov 03 20:32:46.548653 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.548516    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9" event={"ID":"7eefc668-8d72-4ee2-b047-91d13318ec49","Type":"ContainerDied","Data":"673af302896cae623bf6a3dbac75768776ba55d92075ca9f7eaa336e77bfec86"}
Nov 03 20:32:46.548653 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.548531    4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="673af302896cae623bf6a3dbac75768776ba55d92075ca9f7eaa336e77bfec86"
Nov 03 20:32:46.550709 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.550688    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:32:46.617352 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.617320    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:32:46.637519 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.637479    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-envoy\") pod \"7eefc668-8d72-4ee2-b047-91d13318ec49\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") "
Nov 03 20:32:46.637727 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.637639    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eefc668-8d72-4ee2-b047-91d13318ec49-kube-api-access-tsdrw\" (UniqueName: \"kubernetes.io/projected/7eefc668-8d72-4ee2-b047-91d13318ec49-kube-api-access-tsdrw\") pod \"7eefc668-8d72-4ee2-b047-91d13318ec49\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") "
Nov 03 20:32:46.637727 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.637706    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-data\") pod \"7eefc668-8d72-4ee2-b047-91d13318ec49\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") "
Nov 03 20:32:46.637876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.637769    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-workload-certs\") pod \"7eefc668-8d72-4ee2-b047-91d13318ec49\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") "
Nov 03 20:32:46.637876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.637802    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-credential-socket\") pod \"7eefc668-8d72-4ee2-b047-91d13318ec49\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") "
Nov 03 20:32:46.638294 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.638019    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-workload-socket\") pod \"7eefc668-8d72-4ee2-b047-91d13318ec49\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") "
Nov 03 20:32:46.638294 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.638097    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-podinfo\") pod \"7eefc668-8d72-4ee2-b047-91d13318ec49\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") "
Nov 03 20:32:46.638294 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.638139    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-token\" (UniqueName: \"kubernetes.io/projected/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-token\") pod \"7eefc668-8d72-4ee2-b047-91d13318ec49\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") "
Nov 03 20:32:46.638294 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.638193    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eefc668-8d72-4ee2-b047-91d13318ec49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eefc668-8d72-4ee2-b047-91d13318ec49-istiod-ca-cert\") pod \"7eefc668-8d72-4ee2-b047-91d13318ec49\" (UID: \"7eefc668-8d72-4ee2-b047-91d13318ec49\") "
Nov 03 20:32:46.638580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.638367    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-credential-socket" pod "7eefc668-8d72-4ee2-b047-91d13318ec49" (UID: "7eefc668-8d72-4ee2-b047-91d13318ec49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:46.638580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.638446    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-workload-socket" pod "7eefc668-8d72-4ee2-b047-91d13318ec49" (UID: "7eefc668-8d72-4ee2-b047-91d13318ec49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:46.638806 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.638767    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eefc668-8d72-4ee2-b047-91d13318ec49-istiod-ca-cert" pod "7eefc668-8d72-4ee2-b047-91d13318ec49" (UID: "7eefc668-8d72-4ee2-b047-91d13318ec49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:46.638806 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.638774    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-data" pod "7eefc668-8d72-4ee2-b047-91d13318ec49" (UID: "7eefc668-8d72-4ee2-b047-91d13318ec49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:46.638984 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.638918    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-workload-certs" pod "7eefc668-8d72-4ee2-b047-91d13318ec49" (UID: "7eefc668-8d72-4ee2-b047-91d13318ec49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:46.639193 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.639088    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:46.639193 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.639101    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eefc668-8d72-4ee2-b047-91d13318ec49-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:46.639193 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.639111    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:46.639193 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.639119    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:46.639193 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.639128    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:46.641937 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.641911    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-token" pod "7eefc668-8d72-4ee2-b047-91d13318ec49" (UID: "7eefc668-8d72-4ee2-b047-91d13318ec49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:46.651984 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.651942    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-podinfo" pod "7eefc668-8d72-4ee2-b047-91d13318ec49" (UID: "7eefc668-8d72-4ee2-b047-91d13318ec49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:32:46.651984 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.651978    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eefc668-8d72-4ee2-b047-91d13318ec49-kube-api-access-tsdrw" pod "7eefc668-8d72-4ee2-b047-91d13318ec49" (UID: "7eefc668-8d72-4ee2-b047-91d13318ec49"). InnerVolumeSpecName "kube-api-access-tsdrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:46.652542 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.652501    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-envoy" pod "7eefc668-8d72-4ee2-b047-91d13318ec49" (UID: "7eefc668-8d72-4ee2-b047-91d13318ec49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:46.740480 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.740394    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-workload-socket\") pod \"61f598b0-63c5-449a-b963-3d9921553aae\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") "
Nov 03 20:32:46.740705 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.740499    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f598b0-63c5-449a-b963-3d9921553aae-kube-api-access-d2854\" (UniqueName: \"kubernetes.io/projected/61f598b0-63c5-449a-b963-3d9921553aae-kube-api-access-d2854\") pod \"61f598b0-63c5-449a-b963-3d9921553aae\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") "
Nov 03 20:32:46.740705 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.740665    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-istio-envoy\") pod \"61f598b0-63c5-449a-b963-3d9921553aae\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") "
Nov 03 20:32:46.740848 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.740710    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-workload-socket" pod "61f598b0-63c5-449a-b963-3d9921553aae" (UID: "61f598b0-63c5-449a-b963-3d9921553aae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:46.740848 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.740742    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f598b0-63c5-449a-b963-3d9921553aae-istio-token\" (UniqueName: \"kubernetes.io/projected/61f598b0-63c5-449a-b963-3d9921553aae-istio-token\") pod \"61f598b0-63c5-449a-b963-3d9921553aae\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") "
Nov 03 20:32:46.740848 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.740786    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61f598b0-63c5-449a-b963-3d9921553aae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f598b0-63c5-449a-b963-3d9921553aae-istio-podinfo\") pod \"61f598b0-63c5-449a-b963-3d9921553aae\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") "
Nov 03 20:32:46.740848 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.740812    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61f598b0-63c5-449a-b963-3d9921553aae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f598b0-63c5-449a-b963-3d9921553aae-istiod-ca-cert\") pod \"61f598b0-63c5-449a-b963-3d9921553aae\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") "
Nov 03 20:32:46.741047 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.740911    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-workload-certs\") pod \"61f598b0-63c5-449a-b963-3d9921553aae\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") "
Nov 03 20:32:46.741047 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.741022    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-istio-data\") pod \"61f598b0-63c5-449a-b963-3d9921553aae\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") "
Nov 03 20:32:46.741158 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.741127    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-credential-socket\") pod \"61f598b0-63c5-449a-b963-3d9921553aae\" (UID: \"61f598b0-63c5-449a-b963-3d9921553aae\") "
Nov 03 20:32:46.741219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.741177    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f598b0-63c5-449a-b963-3d9921553aae-istiod-ca-cert" pod "61f598b0-63c5-449a-b963-3d9921553aae" (UID: "61f598b0-63c5-449a-b963-3d9921553aae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:46.741352 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.741330    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-workload-certs" pod "61f598b0-63c5-449a-b963-3d9921553aae" (UID: "61f598b0-63c5-449a-b963-3d9921553aae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:46.741556 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.741531    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-istio-data" pod "61f598b0-63c5-449a-b963-3d9921553aae" (UID: "61f598b0-63c5-449a-b963-3d9921553aae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:46.741556 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.741542    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-credential-socket" pod "61f598b0-63c5-449a-b963-3d9921553aae" (UID: "61f598b0-63c5-449a-b963-3d9921553aae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:46.741963 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.741940    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:46.742100 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.741966    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f598b0-63c5-449a-b963-3d9921553aae-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:46.742100 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.741980    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsdrw\" (UniqueName: \"kubernetes.io/projected/7eefc668-8d72-4ee2-b047-91d13318ec49-kube-api-access-tsdrw\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:46.742100 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.741993    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:46.742100 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.742005    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:46.742100 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.742018    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:46.742100 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.742032    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:46.742100 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.742044    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:46.742100 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.742058    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eefc668-8d72-4ee2-b047-91d13318ec49-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:46.743524 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.743504    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f598b0-63c5-449a-b963-3d9921553aae-kube-api-access-d2854" pod "61f598b0-63c5-449a-b963-3d9921553aae" (UID: "61f598b0-63c5-449a-b963-3d9921553aae"). InnerVolumeSpecName "kube-api-access-d2854". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:46.743622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.743534    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f598b0-63c5-449a-b963-3d9921553aae-istio-token" pod "61f598b0-63c5-449a-b963-3d9921553aae" (UID: "61f598b0-63c5-449a-b963-3d9921553aae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:46.743693 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.743670    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-istio-envoy" pod "61f598b0-63c5-449a-b963-3d9921553aae" (UID: "61f598b0-63c5-449a-b963-3d9921553aae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:46.743733 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.743687    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61f598b0-63c5-449a-b963-3d9921553aae-istio-podinfo" pod "61f598b0-63c5-449a-b963-3d9921553aae" (UID: "61f598b0-63c5-449a-b963-3d9921553aae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:32:46.844435 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.844404    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f598b0-63c5-449a-b963-3d9921553aae-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:46.844435 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.844431    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f598b0-63c5-449a-b963-3d9921553aae-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:46.844435 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.844441    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f598b0-63c5-449a-b963-3d9921553aae-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:46.844683 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:46.844452    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2854\" (UniqueName: \"kubernetes.io/projected/61f598b0-63c5-449a-b963-3d9921553aae-kube-api-access-d2854\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:47.557358 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:47.557261    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"
Nov 03 20:32:47.557358 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:47.557293    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7" event={"ID":"61f598b0-63c5-449a-b963-3d9921553aae","Type":"ContainerDied","Data":"af4ff79520b876e3ced132e310d95ac8eb7f6bdd1eba282f9f93f16c96b185f7"}
Nov 03 20:32:47.557931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:47.557369    4421 scope.go:122] "RemoveContainer" containerID="d017ce5afee266d337a066c74c14643a2fedfdeac2a3d5b1b85406aed27511fb"
Nov 03 20:32:47.557931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:47.557456    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"
Nov 03 20:32:47.587326 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:47.587268    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"]
Nov 03 20:32:47.594176 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:47.594152    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-hqdq9"]
Nov 03 20:32:47.607112 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:47.607089    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"]
Nov 03 20:32:47.612430 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:47.612411    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-p2vp7"]
Nov 03 20:32:48.103478 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:48.103438    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61f598b0-63c5-449a-b963-3d9921553aae" path="/var/lib/kubelet/pods/61f598b0-63c5-449a-b963-3d9921553aae/volumes"
Nov 03 20:32:48.105746 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:48.105719    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eefc668-8d72-4ee2-b047-91d13318ec49" path="/var/lib/kubelet/pods/7eefc668-8d72-4ee2-b047-91d13318ec49/volumes"
Nov 03 20:32:49.630935 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:32:49.630901    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:32:50.938098 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:50.938064    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs"]
Nov 03 20:32:50.939368 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:50.939345    4421 cpu_manager.go:414] "containerMap: removing container" podUID="7eefc668-8d72-4ee2-b047-91d13318ec49" containerName="istio-proxy"
Nov 03 20:32:50.939481 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:50.939369    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eefc668-8d72-4ee2-b047-91d13318ec49" containerName="istio-proxy"
Nov 03 20:32:50.939481 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:50.939394    4421 cpu_manager.go:414] "containerMap: removing container" podUID="61f598b0-63c5-449a-b963-3d9921553aae" containerName="istio-proxy"
Nov 03 20:32:50.939481 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:50.939402    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f598b0-63c5-449a-b963-3d9921553aae" containerName="istio-proxy"
Nov 03 20:32:50.939684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:50.939669    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eefc668-8d72-4ee2-b047-91d13318ec49" containerName="istio-proxy"
Nov 03 20:32:50.939743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:50.939690    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f598b0-63c5-449a-b963-3d9921553aae" containerName="istio-proxy"
Nov 03 20:32:50.946526 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:50.946503    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs"
Nov 03 20:32:50.971949 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:50.971887    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-05473b-serving-cert\""
Nov 03 20:32:50.979125 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:50.978577    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs"]
Nov 03 20:32:51.067012 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:51.066968    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabfcd88-38cc-44ae-bb3c-df42c13f0684-metrics-certs\") pod \"maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs\" (UID: \"eabfcd88-38cc-44ae-bb3c-df42c13f0684\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs"
Nov 03 20:32:51.067228 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:51.067182    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz9b5\" (UniqueName: \"kubernetes.io/projected/eabfcd88-38cc-44ae-bb3c-df42c13f0684-kube-api-access-rz9b5\") pod \"maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs\" (UID: \"eabfcd88-38cc-44ae-bb3c-df42c13f0684\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs"
Nov 03 20:32:51.067444 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:51.067420    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabfcd88-38cc-44ae-bb3c-df42c13f0684-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs\" (UID: \"eabfcd88-38cc-44ae-bb3c-df42c13f0684\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs"
Nov 03 20:32:51.067612 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:51.067573    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eabfcd88-38cc-44ae-bb3c-df42c13f0684-maas-api-tls\") pod \"maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs\" (UID: \"eabfcd88-38cc-44ae-bb3c-df42c13f0684\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs"
Nov 03 20:32:51.185386 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:51.185339    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz9b5\" (UniqueName: \"kubernetes.io/projected/eabfcd88-38cc-44ae-bb3c-df42c13f0684-kube-api-access-rz9b5\") pod \"maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs\" (UID: \"eabfcd88-38cc-44ae-bb3c-df42c13f0684\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs"
Nov 03 20:32:51.186226 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:51.186197    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabfcd88-38cc-44ae-bb3c-df42c13f0684-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs\" (UID: \"eabfcd88-38cc-44ae-bb3c-df42c13f0684\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs"
Nov 03 20:32:51.186507 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:51.186489    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eabfcd88-38cc-44ae-bb3c-df42c13f0684-maas-api-tls\") pod \"maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs\" (UID: \"eabfcd88-38cc-44ae-bb3c-df42c13f0684\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs"
Nov 03 20:32:51.186753 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:51.186733    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabfcd88-38cc-44ae-bb3c-df42c13f0684-metrics-certs\") pod \"maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs\" (UID: \"eabfcd88-38cc-44ae-bb3c-df42c13f0684\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs"
Nov 03 20:32:51.187580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:51.187496    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabfcd88-38cc-44ae-bb3c-df42c13f0684-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs\" (UID: \"eabfcd88-38cc-44ae-bb3c-df42c13f0684\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs"
Nov 03 20:32:51.232711 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:51.225559    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eabfcd88-38cc-44ae-bb3c-df42c13f0684-maas-api-tls\") pod \"maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs\" (UID: \"eabfcd88-38cc-44ae-bb3c-df42c13f0684\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs"
Nov 03 20:32:51.275623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:51.269282    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabfcd88-38cc-44ae-bb3c-df42c13f0684-metrics-certs\") pod \"maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs\" (UID: \"eabfcd88-38cc-44ae-bb3c-df42c13f0684\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs"
Nov 03 20:32:51.275623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:51.272180    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz9b5\" (UniqueName: \"kubernetes.io/projected/eabfcd88-38cc-44ae-bb3c-df42c13f0684-kube-api-access-rz9b5\") pod \"maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs\" (UID: \"eabfcd88-38cc-44ae-bb3c-df42c13f0684\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs"
Nov 03 20:32:51.302013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:51.294557    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs"
Nov 03 20:32:51.830625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:51.830515    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs"]
Nov 03 20:32:51.880660 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:32:51.870234    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeabfcd88_38cc_44ae_bb3c_df42c13f0684.slice/crio-fc880127a5b1bed6f4d8d228d771c14d06416f0f14c0cbb2ed8c89e6f4165f44 WatchSource:0}: Error finding container fc880127a5b1bed6f4d8d228d771c14d06416f0f14c0cbb2ed8c89e6f4165f44: Status 404 returned error can't find the container with id fc880127a5b1bed6f4d8d228d771c14d06416f0f14c0cbb2ed8c89e6f4165f44
Nov 03 20:32:52.606241 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:52.606202    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs" event={"ID":"eabfcd88-38cc-44ae-bb3c-df42c13f0684","Type":"ContainerStarted","Data":"64e286545e35b9588f2b45781c356198ed4bfb901e82b9e9e393cab0a7d08c3a"}
Nov 03 20:32:52.606701 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:52.606257    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs" event={"ID":"eabfcd88-38cc-44ae-bb3c-df42c13f0684","Type":"ContainerStarted","Data":"fc880127a5b1bed6f4d8d228d771c14d06416f0f14c0cbb2ed8c89e6f4165f44"}
Nov 03 20:32:52.606701 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:52.606331    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs"
Nov 03 20:32:52.629063 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:52.629018    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs" podStartSLOduration=2.109266286 podStartE2EDuration="2.629004904s" totalImagesPullingTime="519.738618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:52.625879273 +0000 UTC m=+1902.759007736" watchObservedRunningTime="2026-11-03 20:32:52.629004904 +0000 UTC m=+1902.762133366"
Nov 03 20:32:58.798623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:32:58.784387    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs"
Nov 03 20:33:07.244770 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:07.244719    4421 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:33:07.457701 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:07.244801    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr" podUID="4fbddfde-9453-458d-b50b-05a3deeabf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:33:08.209392 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:08.209358    4421 generic.go:356] "Generic (PLEG): container finished" podID="4fbddfde-9453-458d-b50b-05a3deeabf74" containerID="805470c43d8e2f5741e50fc54f9579d00ba1df5ffcefb0db8d3d68d1197816ef" exitCode=0
Nov 03 20:33:08.209618 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:08.209436    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr" event={"ID":"4fbddfde-9453-458d-b50b-05a3deeabf74","Type":"ContainerDied","Data":"805470c43d8e2f5741e50fc54f9579d00ba1df5ffcefb0db8d3d68d1197816ef"}
Nov 03 20:33:08.210026 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:08.210008    4421 scope.go:122] "RemoveContainer" containerID="805470c43d8e2f5741e50fc54f9579d00ba1df5ffcefb0db8d3d68d1197816ef"
Nov 03 20:33:08.212291 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:08.212260    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:33:08.212402 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:08.212334    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:33:08.212402 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:08.212371    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:33:09.244646 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:09.244145    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr" event={"ID":"4fbddfde-9453-458d-b50b-05a3deeabf74","Type":"ContainerStarted","Data":"4c788c30b6bfa3f8c05cc5d936aadd6b9b35a439722bb8d5a71bf38b7a2fc82d"}
Nov 03 20:33:10.165685 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:10.165150    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:33:10.165685 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:10.165283    4421 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 03 20:33:10.165685 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:10.165341    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr" podUID="4fbddfde-9453-458d-b50b-05a3deeabf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 03 20:33:11.173450 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:11.173397    4421 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 03 20:33:11.174089 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:11.174048    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr" podUID="4fbddfde-9453-458d-b50b-05a3deeabf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 03 20:33:12.165014 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:12.164974    4421 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 03 20:33:12.165321 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:12.165291    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr" podUID="4fbddfde-9453-458d-b50b-05a3deeabf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 03 20:33:12.222268 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:12.222233    4421 scope.go:122] "RemoveContainer" containerID="b01881ec6dfe12f1dbc37fff50f48ad859e9531b5e004f87d0c831acae2a64dd"
Nov 03 20:33:13.175557 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:13.175500    4421 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 03 20:33:13.175895 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:13.175861    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr" podUID="4fbddfde-9453-458d-b50b-05a3deeabf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 03 20:33:14.164624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:14.163928    4421 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 03 20:33:14.164624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:14.163991    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr" podUID="4fbddfde-9453-458d-b50b-05a3deeabf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 03 20:33:14.894551 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:14.894516    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8"]
Nov 03 20:33:14.899880 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:14.899857    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8"
Nov 03 20:33:14.909984 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:14.909959    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"agc-cred-0d43e4ffb5b7\""
Nov 03 20:33:14.910169 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:14.910040    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"payload-processing-external-model-e2e--f590b3a5-dockercfg-mjjk7\""
Nov 03 20:33:14.910667 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:14.910635    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8"]
Nov 03 20:33:14.914729 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:14.914705    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"openshift-service-ca.crt\""
Nov 03 20:33:14.914850 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:14.914791    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"kube-root-ca.crt\""
Nov 03 20:33:14.914917 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:14.914896    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-05473b\""
Nov 03 20:33:14.915527 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:14.915510    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"routing-overlay\""
Nov 03 20:33:14.978013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:14.977969    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e81b052-3cb9-47cf-9b09-67341b4deee1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8\" (UID: \"6e81b052-3cb9-47cf-9b09-67341b4deee1\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8"
Nov 03 20:33:14.978252 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:14.978072    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e81b052-3cb9-47cf-9b09-67341b4deee1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8\" (UID: \"6e81b052-3cb9-47cf-9b09-67341b4deee1\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8"
Nov 03 20:33:14.978252 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:14.978166    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hhck\" (UniqueName: \"kubernetes.io/projected/6e81b052-3cb9-47cf-9b09-67341b4deee1-kube-api-access-8hhck\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8\" (UID: \"6e81b052-3cb9-47cf-9b09-67341b4deee1\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8"
Nov 03 20:33:14.978474 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:14.978364    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e81b052-3cb9-47cf-9b09-67341b4deee1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8\" (UID: \"6e81b052-3cb9-47cf-9b09-67341b4deee1\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8"
Nov 03 20:33:15.105653 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:15.105615    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hhck\" (UniqueName: \"kubernetes.io/projected/6e81b052-3cb9-47cf-9b09-67341b4deee1-kube-api-access-8hhck\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8\" (UID: \"6e81b052-3cb9-47cf-9b09-67341b4deee1\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8"
Nov 03 20:33:15.163228 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:15.129888    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e81b052-3cb9-47cf-9b09-67341b4deee1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8\" (UID: \"6e81b052-3cb9-47cf-9b09-67341b4deee1\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8"
Nov 03 20:33:15.192641 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:15.167826    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e81b052-3cb9-47cf-9b09-67341b4deee1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8\" (UID: \"6e81b052-3cb9-47cf-9b09-67341b4deee1\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8"
Nov 03 20:33:15.193418 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:15.193388    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e81b052-3cb9-47cf-9b09-67341b4deee1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8\" (UID: \"6e81b052-3cb9-47cf-9b09-67341b4deee1\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8"
Nov 03 20:33:15.319581 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:15.188169    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e81b052-3cb9-47cf-9b09-67341b4deee1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8\" (UID: \"6e81b052-3cb9-47cf-9b09-67341b4deee1\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8"
Nov 03 20:33:15.319891 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:15.187345    4421 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 03 20:33:15.320842 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:15.208404    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e81b052-3cb9-47cf-9b09-67341b4deee1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8\" (UID: \"6e81b052-3cb9-47cf-9b09-67341b4deee1\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8"
Nov 03 20:33:15.322591 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:15.299359    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e81b052-3cb9-47cf-9b09-67341b4deee1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8\" (UID: \"6e81b052-3cb9-47cf-9b09-67341b4deee1\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8"
Nov 03 20:33:15.322832 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:15.306101    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hhck\" (UniqueName: \"kubernetes.io/projected/6e81b052-3cb9-47cf-9b09-67341b4deee1-kube-api-access-8hhck\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8\" (UID: \"6e81b052-3cb9-47cf-9b09-67341b4deee1\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8"
Nov 03 20:33:15.323012 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:15.320095    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr" podUID="4fbddfde-9453-458d-b50b-05a3deeabf74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 03 20:33:15.524251 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:15.524200    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8"
Nov 03 20:33:15.820964 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:33:15.820920    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e81b052_3cb9_47cf_9b09_67341b4deee1.slice/crio-737e922c3824c696920575cf7a00190e26c0fc9a266fd50b4423391f078750f7 WatchSource:0}: Error finding container 737e922c3824c696920575cf7a00190e26c0fc9a266fd50b4423391f078750f7: Status 404 returned error can't find the container with id 737e922c3824c696920575cf7a00190e26c0fc9a266fd50b4423391f078750f7
Nov 03 20:33:15.821424 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:15.821362    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8"]
Nov 03 20:33:16.136810 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:16.136728    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j"]
Nov 03 20:33:16.140584 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:16.140564    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j"
Nov 03 20:33:16.149179 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:16.149157    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-d8d0fa\""
Nov 03 20:33:16.152870 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:16.152843    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j"]
Nov 03 20:33:16.174835 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:16.174802    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:33:16.175367 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:16.175345    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:33:16.177572 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:16.177549    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:33:16.235182 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:16.235145    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cdc9b55-921d-4ece-baae-1134aa71f5f7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j\" (UID: \"3cdc9b55-921d-4ece-baae-1134aa71f5f7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j"
Nov 03 20:33:16.236505 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:16.235446    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t88bt\" (UniqueName: \"kubernetes.io/projected/3cdc9b55-921d-4ece-baae-1134aa71f5f7-kube-api-access-t88bt\") pod \"payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j\" (UID: \"3cdc9b55-921d-4ece-baae-1134aa71f5f7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j"
Nov 03 20:33:16.337362 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:16.337326    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t88bt\" (UniqueName: \"kubernetes.io/projected/3cdc9b55-921d-4ece-baae-1134aa71f5f7-kube-api-access-t88bt\") pod \"payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j\" (UID: \"3cdc9b55-921d-4ece-baae-1134aa71f5f7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j"
Nov 03 20:33:16.337788 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:16.337766    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cdc9b55-921d-4ece-baae-1134aa71f5f7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j\" (UID: \"3cdc9b55-921d-4ece-baae-1134aa71f5f7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j"
Nov 03 20:33:16.338980 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:16.338953    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cdc9b55-921d-4ece-baae-1134aa71f5f7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j\" (UID: \"3cdc9b55-921d-4ece-baae-1134aa71f5f7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j"
Nov 03 20:33:16.401342 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:16.394459    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t88bt\" (UniqueName: \"kubernetes.io/projected/3cdc9b55-921d-4ece-baae-1134aa71f5f7-kube-api-access-t88bt\") pod \"payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j\" (UID: \"3cdc9b55-921d-4ece-baae-1134aa71f5f7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j"
Nov 03 20:33:16.517626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:16.514907    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j"
Nov 03 20:33:16.738645 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:16.737739    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8" event={"ID":"6e81b052-3cb9-47cf-9b09-67341b4deee1","Type":"ContainerStarted","Data":"7db6b0d4b7be8d549a6d82b05afb80cbb6058e5f26879cbc322234f5122c302c"}
Nov 03 20:33:16.738645 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:16.737826    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8" event={"ID":"6e81b052-3cb9-47cf-9b09-67341b4deee1","Type":"ContainerStarted","Data":"737e922c3824c696920575cf7a00190e26c0fc9a266fd50b4423391f078750f7"}
Nov 03 20:33:16.738645 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:16.737856    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8"
Nov 03 20:33:16.822050 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:16.822002    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8" podStartSLOduration=2.821984411 podStartE2EDuration="2.821984411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:16.819842799 +0000 UTC m=+1926.952971262" watchObservedRunningTime="2026-11-03 20:33:16.821984411 +0000 UTC m=+1926.955112873"
Nov 03 20:33:17.049257 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:17.049217    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j"]
Nov 03 20:33:17.052763 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:33:17.052732    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cdc9b55_921d_4ece_baae_1134aa71f5f7.slice/crio-1fb2beaba32181efbe45fe5833062b9c894ee7c908e3153c4741aa27ae831d5c WatchSource:0}: Error finding container 1fb2beaba32181efbe45fe5833062b9c894ee7c908e3153c4741aa27ae831d5c: Status 404 returned error can't find the container with id 1fb2beaba32181efbe45fe5833062b9c894ee7c908e3153c4741aa27ae831d5c
Nov 03 20:33:17.746442 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:17.746405    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j" event={"ID":"3cdc9b55-921d-4ece-baae-1134aa71f5f7","Type":"ContainerStarted","Data":"d75b70b409019d5a056bd93713b3a08846b148442340e69a711db4fbb06442ba"}
Nov 03 20:33:17.746890 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:17.746491    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j" event={"ID":"3cdc9b55-921d-4ece-baae-1134aa71f5f7","Type":"ContainerStarted","Data":"1fb2beaba32181efbe45fe5833062b9c894ee7c908e3153c4741aa27ae831d5c"}
Nov 03 20:33:17.766379 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:17.766335    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j" podStartSLOduration=1.766322113 podStartE2EDuration="1.766322113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:17.76563683 +0000 UTC m=+1927.898765292" watchObservedRunningTime="2026-11-03 20:33:17.766322113 +0000 UTC m=+1927.899450574"
Nov 03 20:33:18.755094 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:18.755053    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j"
Nov 03 20:33:19.632044 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:33:19.631964    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:33:20.113850 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:20.113818    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8"]
Nov 03 20:33:20.120664 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:20.120639    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8"
Nov 03 20:33:20.125133 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:20.125086    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-421604-serving-cert\""
Nov 03 20:33:20.133502 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:20.133477    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8"]
Nov 03 20:33:20.191638 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:20.191581    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02e2fd7-7327-4777-ba31-4c683a809719-maas-api-tls\") pod \"maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8\" (UID: \"b02e2fd7-7327-4777-ba31-4c683a809719\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8"
Nov 03 20:33:20.191845 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:20.191752    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b02e2fd7-7327-4777-ba31-4c683a809719-metrics-certs\") pod \"maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8\" (UID: \"b02e2fd7-7327-4777-ba31-4c683a809719\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8"
Nov 03 20:33:20.191845 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:20.191810    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbwwp\" (UniqueName: \"kubernetes.io/projected/b02e2fd7-7327-4777-ba31-4c683a809719-kube-api-access-fbwwp\") pod \"maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8\" (UID: \"b02e2fd7-7327-4777-ba31-4c683a809719\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8"
Nov 03 20:33:20.191971 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:20.191911    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02e2fd7-7327-4777-ba31-4c683a809719-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8\" (UID: \"b02e2fd7-7327-4777-ba31-4c683a809719\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8"
Nov 03 20:33:20.296251 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:20.296204    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02e2fd7-7327-4777-ba31-4c683a809719-maas-api-tls\") pod \"maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8\" (UID: \"b02e2fd7-7327-4777-ba31-4c683a809719\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8"
Nov 03 20:33:20.296571 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:20.296300    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b02e2fd7-7327-4777-ba31-4c683a809719-metrics-certs\") pod \"maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8\" (UID: \"b02e2fd7-7327-4777-ba31-4c683a809719\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8"
Nov 03 20:33:20.296571 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:20.296349    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbwwp\" (UniqueName: \"kubernetes.io/projected/b02e2fd7-7327-4777-ba31-4c683a809719-kube-api-access-fbwwp\") pod \"maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8\" (UID: \"b02e2fd7-7327-4777-ba31-4c683a809719\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8"
Nov 03 20:33:20.296571 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:20.296387    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02e2fd7-7327-4777-ba31-4c683a809719-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8\" (UID: \"b02e2fd7-7327-4777-ba31-4c683a809719\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8"
Nov 03 20:33:20.297383 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:20.297356    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02e2fd7-7327-4777-ba31-4c683a809719-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8\" (UID: \"b02e2fd7-7327-4777-ba31-4c683a809719\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8"
Nov 03 20:33:20.297855 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:33:20.297786    4421 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-serving-cert: secret "maas-api-e2e-extmod-b-421604-serving-cert" not found
Nov 03 20:33:20.297855 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:33:20.297851    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b02e2fd7-7327-4777-ba31-4c683a809719-maas-api-tls podName:b02e2fd7-7327-4777-ba31-4c683a809719 nodeName:}" failed. No retries permitted until 2026-11-03 20:33:20.797833355 +0000 UTC m=+1930.930961816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b02e2fd7-7327-4777-ba31-4c683a809719-maas-api-tls") pod "maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8" (UID: "b02e2fd7-7327-4777-ba31-4c683a809719") : secret "maas-api-e2e-extmod-b-421604-serving-cert" not found
Nov 03 20:33:20.354658 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:20.347888    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b02e2fd7-7327-4777-ba31-4c683a809719-metrics-certs\") pod \"maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8\" (UID: \"b02e2fd7-7327-4777-ba31-4c683a809719\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8"
Nov 03 20:33:20.354658 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:20.349894    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbwwp\" (UniqueName: \"kubernetes.io/projected/b02e2fd7-7327-4777-ba31-4c683a809719-kube-api-access-fbwwp\") pod \"maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8\" (UID: \"b02e2fd7-7327-4777-ba31-4c683a809719\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8"
Nov 03 20:33:20.833624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:20.820512    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02e2fd7-7327-4777-ba31-4c683a809719-maas-api-tls\") pod \"maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8\" (UID: \"b02e2fd7-7327-4777-ba31-4c683a809719\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8"
Nov 03 20:33:20.870693 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:20.868870    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02e2fd7-7327-4777-ba31-4c683a809719-maas-api-tls\") pod \"maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8\" (UID: \"b02e2fd7-7327-4777-ba31-4c683a809719\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8"
Nov 03 20:33:21.056231 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:21.056198    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8"
Nov 03 20:33:21.496377 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:21.496341    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8"]
Nov 03 20:33:21.516499 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:33:21.516466    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02e2fd7_7327_4777_ba31_4c683a809719.slice/crio-e5a55207e65ae8d428c4e9bd68f308295f944677f3ff764854ab5dcfc6deb9ce WatchSource:0}: Error finding container e5a55207e65ae8d428c4e9bd68f308295f944677f3ff764854ab5dcfc6deb9ce: Status 404 returned error can't find the container with id e5a55207e65ae8d428c4e9bd68f308295f944677f3ff764854ab5dcfc6deb9ce
Nov 03 20:33:21.861167 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:21.861074    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8" event={"ID":"b02e2fd7-7327-4777-ba31-4c683a809719","Type":"ContainerStarted","Data":"e5a55207e65ae8d428c4e9bd68f308295f944677f3ff764854ab5dcfc6deb9ce"}
Nov 03 20:33:22.870427 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:22.870380    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8" event={"ID":"b02e2fd7-7327-4777-ba31-4c683a809719","Type":"ContainerStarted","Data":"79fcd906f100eb3c80b8060a836220a1fb9bab4124eb725ccd789a55a62f8d46"}
Nov 03 20:33:22.870881 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:22.870492    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8"
Nov 03 20:33:22.892508 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:22.892463    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8" podStartSLOduration=2.43848681 podStartE2EDuration="2.892450019s" totalImagesPullingTime="453.963209ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:22.88958922 +0000 UTC m=+1933.022717696" watchObservedRunningTime="2026-11-03 20:33:22.892450019 +0000 UTC m=+1933.025578480"
Nov 03 20:33:27.749439 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:27.749404    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8"
Nov 03 20:33:28.883928 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:28.883899    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8"
Nov 03 20:33:29.888671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:29.877863    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j"
Nov 03 20:33:30.618380 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:30.618343    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7"]
Nov 03 20:33:30.629671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:30.625523    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7"
Nov 03 20:33:30.652634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:30.649115    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-d8d0fa-serving-cert\""
Nov 03 20:33:30.661937 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:30.661755    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7"]
Nov 03 20:33:30.753653 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:30.753577    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m488\" (UniqueName: \"kubernetes.io/projected/024fea96-e087-4d81-b1f1-1801c4ecf0ab-kube-api-access-6m488\") pod \"maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7\" (UID: \"024fea96-e087-4d81-b1f1-1801c4ecf0ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7"
Nov 03 20:33:30.753941 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:30.753920    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/024fea96-e087-4d81-b1f1-1801c4ecf0ab-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7\" (UID: \"024fea96-e087-4d81-b1f1-1801c4ecf0ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7"
Nov 03 20:33:30.753989 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:30.753954    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/024fea96-e087-4d81-b1f1-1801c4ecf0ab-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7\" (UID: \"024fea96-e087-4d81-b1f1-1801c4ecf0ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7"
Nov 03 20:33:30.754070 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:30.754059    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/024fea96-e087-4d81-b1f1-1801c4ecf0ab-metrics-certs\") pod \"maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7\" (UID: \"024fea96-e087-4d81-b1f1-1801c4ecf0ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7"
Nov 03 20:33:30.855421 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:30.855388    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/024fea96-e087-4d81-b1f1-1801c4ecf0ab-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7\" (UID: \"024fea96-e087-4d81-b1f1-1801c4ecf0ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7"
Nov 03 20:33:30.855654 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:30.855429    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/024fea96-e087-4d81-b1f1-1801c4ecf0ab-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7\" (UID: \"024fea96-e087-4d81-b1f1-1801c4ecf0ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7"
Nov 03 20:33:30.855654 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:30.855485    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/024fea96-e087-4d81-b1f1-1801c4ecf0ab-metrics-certs\") pod \"maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7\" (UID: \"024fea96-e087-4d81-b1f1-1801c4ecf0ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7"
Nov 03 20:33:30.855654 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:30.855557    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m488\" (UniqueName: \"kubernetes.io/projected/024fea96-e087-4d81-b1f1-1801c4ecf0ab-kube-api-access-6m488\") pod \"maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7\" (UID: \"024fea96-e087-4d81-b1f1-1801c4ecf0ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7"
Nov 03 20:33:30.856064 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:30.856042    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/024fea96-e087-4d81-b1f1-1801c4ecf0ab-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7\" (UID: \"024fea96-e087-4d81-b1f1-1801c4ecf0ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7"
Nov 03 20:33:30.858167 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:30.858142    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/024fea96-e087-4d81-b1f1-1801c4ecf0ab-metrics-certs\") pod \"maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7\" (UID: \"024fea96-e087-4d81-b1f1-1801c4ecf0ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7"
Nov 03 20:33:30.858286 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:30.858269    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/024fea96-e087-4d81-b1f1-1801c4ecf0ab-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7\" (UID: \"024fea96-e087-4d81-b1f1-1801c4ecf0ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7"
Nov 03 20:33:30.868457 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:30.868406    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m488\" (UniqueName: \"kubernetes.io/projected/024fea96-e087-4d81-b1f1-1801c4ecf0ab-kube-api-access-6m488\") pod \"maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7\" (UID: \"024fea96-e087-4d81-b1f1-1801c4ecf0ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7"
Nov 03 20:33:30.978634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:30.974963    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7"
Nov 03 20:33:31.932792 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:33:31.932752    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024fea96_e087_4d81_b1f1_1801c4ecf0ab.slice/crio-9f67988c83a142fb9505c2cdf5a0e76b919fa76dbe4f7b8e59dfc6ea76962253 WatchSource:0}: Error finding container 9f67988c83a142fb9505c2cdf5a0e76b919fa76dbe4f7b8e59dfc6ea76962253: Status 404 returned error can't find the container with id 9f67988c83a142fb9505c2cdf5a0e76b919fa76dbe4f7b8e59dfc6ea76962253
Nov 03 20:33:31.932792 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:31.932755    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7"]
Nov 03 20:33:32.410619 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:32.406624    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7" event={"ID":"024fea96-e087-4d81-b1f1-1801c4ecf0ab","Type":"ContainerStarted","Data":"9f67988c83a142fb9505c2cdf5a0e76b919fa76dbe4f7b8e59dfc6ea76962253"}
Nov 03 20:33:33.210623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:33.209998    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q"]
Nov 03 20:33:33.252622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:33.251667    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4"]
Nov 03 20:33:33.425888 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:33.425848    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7" event={"ID":"024fea96-e087-4d81-b1f1-1801c4ecf0ab","Type":"ContainerStarted","Data":"173af2a3050ac782421002cfbc321671a24e91be7aabd4cdd4d281c264673e14"}
Nov 03 20:33:33.426435 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:33.426155    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7"
Nov 03 20:33:33.449424 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:33.449378    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7" podStartSLOduration=2.937842969 podStartE2EDuration="3.449363594s" totalImagesPullingTime="511.520625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:33.447687382 +0000 UTC m=+1943.580815843" watchObservedRunningTime="2026-11-03 20:33:33.449363594 +0000 UTC m=+1943.582492084"
Nov 03 20:33:38.210822 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:38.210777    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q" podUID="a020faaa-cf90-466e-8715-ccf5f987b158" containerName="payload-processing" containerID="cri-o://41a1a73f2362f10f45d0daa2b866817e7899dfcc121b27fd25f94cd8fcc4dcc3" gracePeriod=25
Nov 03 20:33:38.251998 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:38.251954    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4" podUID="68f60111-d601-492f-b82e-9db18a818e3a" containerName="payload-pre-processing" containerID="cri-o://146cdf7f9f9170c6e14ed967ff12200cb645f7df4610a3ee4e12bed3b3ccc805" gracePeriod=25
Nov 03 20:33:38.473230 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:38.473133    4421 generic.go:356] "Generic (PLEG): container finished" podID="68f60111-d601-492f-b82e-9db18a818e3a" containerID="146cdf7f9f9170c6e14ed967ff12200cb645f7df4610a3ee4e12bed3b3ccc805" exitCode=0
Nov 03 20:33:38.473230 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:38.473204    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4" event={"ID":"68f60111-d601-492f-b82e-9db18a818e3a","Type":"ContainerDied","Data":"146cdf7f9f9170c6e14ed967ff12200cb645f7df4610a3ee4e12bed3b3ccc805"}
Nov 03 20:33:38.475390 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:38.475362    4421 generic.go:356] "Generic (PLEG): container finished" podID="a020faaa-cf90-466e-8715-ccf5f987b158" containerID="41a1a73f2362f10f45d0daa2b866817e7899dfcc121b27fd25f94cd8fcc4dcc3" exitCode=0
Nov 03 20:33:38.475559 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:38.475549    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q" event={"ID":"a020faaa-cf90-466e-8715-ccf5f987b158","Type":"ContainerDied","Data":"41a1a73f2362f10f45d0daa2b866817e7899dfcc121b27fd25f94cd8fcc4dcc3"}
Nov 03 20:33:38.567287 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:38.567254    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q"
Nov 03 20:33:38.601805 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:38.601778    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4"
Nov 03 20:33:38.755747 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:38.755669    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68f60111-d601-492f-b82e-9db18a818e3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f60111-d601-492f-b82e-9db18a818e3a-plugins-config-volume\") pod \"68f60111-d601-492f-b82e-9db18a818e3a\" (UID: \"68f60111-d601-492f-b82e-9db18a818e3a\") "
Nov 03 20:33:38.755945 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:38.755763    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a020faaa-cf90-466e-8715-ccf5f987b158-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a020faaa-cf90-466e-8715-ccf5f987b158-plugins-config-volume\") pod \"a020faaa-cf90-466e-8715-ccf5f987b158\" (UID: \"a020faaa-cf90-466e-8715-ccf5f987b158\") "
Nov 03 20:33:38.755945 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:38.755819    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a020faaa-cf90-466e-8715-ccf5f987b158-kube-api-access-vvxbh\" (UniqueName: \"kubernetes.io/projected/a020faaa-cf90-466e-8715-ccf5f987b158-kube-api-access-vvxbh\") pod \"a020faaa-cf90-466e-8715-ccf5f987b158\" (UID: \"a020faaa-cf90-466e-8715-ccf5f987b158\") "
Nov 03 20:33:38.755945 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:38.755853    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f60111-d601-492f-b82e-9db18a818e3a-kube-api-access-txtvw\" (UniqueName: \"kubernetes.io/projected/68f60111-d601-492f-b82e-9db18a818e3a-kube-api-access-txtvw\") pod \"68f60111-d601-492f-b82e-9db18a818e3a\" (UID: \"68f60111-d601-492f-b82e-9db18a818e3a\") "
Nov 03 20:33:38.756146 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:38.756109    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f60111-d601-492f-b82e-9db18a818e3a-plugins-config-volume" pod "68f60111-d601-492f-b82e-9db18a818e3a" (UID: "68f60111-d601-492f-b82e-9db18a818e3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:38.756228 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:38.756203    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a020faaa-cf90-466e-8715-ccf5f987b158-plugins-config-volume" pod "a020faaa-cf90-466e-8715-ccf5f987b158" (UID: "a020faaa-cf90-466e-8715-ccf5f987b158"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:38.756522 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:38.756502    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f60111-d601-492f-b82e-9db18a818e3a-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:38.756586 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:38.756528    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a020faaa-cf90-466e-8715-ccf5f987b158-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:38.758132 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:38.758111    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f60111-d601-492f-b82e-9db18a818e3a-kube-api-access-txtvw" pod "68f60111-d601-492f-b82e-9db18a818e3a" (UID: "68f60111-d601-492f-b82e-9db18a818e3a"). InnerVolumeSpecName "kube-api-access-txtvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:38.758199 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:38.758141    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a020faaa-cf90-466e-8715-ccf5f987b158-kube-api-access-vvxbh" pod "a020faaa-cf90-466e-8715-ccf5f987b158" (UID: "a020faaa-cf90-466e-8715-ccf5f987b158"). InnerVolumeSpecName "kube-api-access-vvxbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:38.861670 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:38.859448    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvxbh\" (UniqueName: \"kubernetes.io/projected/a020faaa-cf90-466e-8715-ccf5f987b158-kube-api-access-vvxbh\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:38.861670 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:38.859483    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txtvw\" (UniqueName: \"kubernetes.io/projected/68f60111-d601-492f-b82e-9db18a818e3a-kube-api-access-txtvw\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:39.439991 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:39.439964    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7"
Nov 03 20:33:39.486445 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:39.486363    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4"
Nov 03 20:33:39.486445 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:39.486390    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4" event={"ID":"68f60111-d601-492f-b82e-9db18a818e3a","Type":"ContainerDied","Data":"d18c863db16d3284a09b5a144708fbeb9c4636a7f3e584d3bb921b833aba04d6"}
Nov 03 20:33:39.486704 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:39.486460    4421 scope.go:122] "RemoveContainer" containerID="146cdf7f9f9170c6e14ed967ff12200cb645f7df4610a3ee4e12bed3b3ccc805"
Nov 03 20:33:39.494003 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:39.493977    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q" event={"ID":"a020faaa-cf90-466e-8715-ccf5f987b158","Type":"ContainerDied","Data":"3675d58ddba6f33bf9fca466c289f04f47acdc6b15675a19893c1bef3e7a8489"}
Nov 03 20:33:39.494163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:39.494087    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q"
Nov 03 20:33:39.508333 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:39.505622    4421 scope.go:122] "RemoveContainer" containerID="41a1a73f2362f10f45d0daa2b866817e7899dfcc121b27fd25f94cd8fcc4dcc3"
Nov 03 20:33:39.525697 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:39.525673    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4"]
Nov 03 20:33:39.537029 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:39.533347    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0f6e09-6dfbd9b95f-5t8r4"]
Nov 03 20:33:39.540683 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:39.540657    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q"]
Nov 03 20:33:39.545030 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:39.545011    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0f6e09-7dcfd6f794-zwx5q"]
Nov 03 20:33:40.018812 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:40.018777    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68f60111-d601-492f-b82e-9db18a818e3a" path="/var/lib/kubelet/pods/68f60111-d601-492f-b82e-9db18a818e3a/volumes"
Nov 03 20:33:40.019273 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:40.019259    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a020faaa-cf90-466e-8715-ccf5f987b158" path="/var/lib/kubelet/pods/a020faaa-cf90-466e-8715-ccf5f987b158/volumes"
Nov 03 20:33:49.629845 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:33:49.629809    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:33:52.135478 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:52.135442    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg"]
Nov 03 20:33:52.161623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:52.154545    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9"]
Nov 03 20:33:57.138435 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:57.136978    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg" podUID="6202aad3-67d4-4cec-8cbc-cdede3f9bb7c" containerName="payload-processing" containerID="cri-o://b03e0696a9d1536263b4dd757936cb855706273e952a478b498bcbf8722974d3" gracePeriod=25
Nov 03 20:33:57.155615 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:57.155539    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9" podUID="f91c8aeb-d184-4b0c-bd49-7297d51cf271" containerName="payload-pre-processing" containerID="cri-o://910c741a7b67da55e14e1c32fd9afa0bb68aa9030872c5f189ce92e17f531316" gracePeriod=25
Nov 03 20:33:57.522809 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:57.522781    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9"
Nov 03 20:33:57.614123 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:57.614075    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f91c8aeb-d184-4b0c-bd49-7297d51cf271-kube-api-access-z22k6\" (UniqueName: \"kubernetes.io/projected/f91c8aeb-d184-4b0c-bd49-7297d51cf271-kube-api-access-z22k6\") pod \"f91c8aeb-d184-4b0c-bd49-7297d51cf271\" (UID: \"f91c8aeb-d184-4b0c-bd49-7297d51cf271\") "
Nov 03 20:33:57.614437 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:57.614418    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f91c8aeb-d184-4b0c-bd49-7297d51cf271-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f91c8aeb-d184-4b0c-bd49-7297d51cf271-plugins-config-volume\") pod \"f91c8aeb-d184-4b0c-bd49-7297d51cf271\" (UID: \"f91c8aeb-d184-4b0c-bd49-7297d51cf271\") "
Nov 03 20:33:57.620975 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:57.620252    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f91c8aeb-d184-4b0c-bd49-7297d51cf271-plugins-config-volume" pod "f91c8aeb-d184-4b0c-bd49-7297d51cf271" (UID: "f91c8aeb-d184-4b0c-bd49-7297d51cf271"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:57.621751 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:57.621717    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f91c8aeb-d184-4b0c-bd49-7297d51cf271-kube-api-access-z22k6" pod "f91c8aeb-d184-4b0c-bd49-7297d51cf271" (UID: "f91c8aeb-d184-4b0c-bd49-7297d51cf271"). InnerVolumeSpecName "kube-api-access-z22k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:57.621895 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:57.621819    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg"
Nov 03 20:33:57.790671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:57.764739    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6202aad3-67d4-4cec-8cbc-cdede3f9bb7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6202aad3-67d4-4cec-8cbc-cdede3f9bb7c-plugins-config-volume\") pod \"6202aad3-67d4-4cec-8cbc-cdede3f9bb7c\" (UID: \"6202aad3-67d4-4cec-8cbc-cdede3f9bb7c\") "
Nov 03 20:33:57.790671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:57.764929    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6202aad3-67d4-4cec-8cbc-cdede3f9bb7c-kube-api-access-6876g\" (UniqueName: \"kubernetes.io/projected/6202aad3-67d4-4cec-8cbc-cdede3f9bb7c-kube-api-access-6876g\") pod \"6202aad3-67d4-4cec-8cbc-cdede3f9bb7c\" (UID: \"6202aad3-67d4-4cec-8cbc-cdede3f9bb7c\") "
Nov 03 20:33:57.790671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:57.765732    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f91c8aeb-d184-4b0c-bd49-7297d51cf271-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:57.790671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:57.773369    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6202aad3-67d4-4cec-8cbc-cdede3f9bb7c-plugins-config-volume" pod "6202aad3-67d4-4cec-8cbc-cdede3f9bb7c" (UID: "6202aad3-67d4-4cec-8cbc-cdede3f9bb7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:57.790671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:57.773443    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z22k6\" (UniqueName: \"kubernetes.io/projected/f91c8aeb-d184-4b0c-bd49-7297d51cf271-kube-api-access-z22k6\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:57.885624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:57.883797    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6202aad3-67d4-4cec-8cbc-cdede3f9bb7c-kube-api-access-6876g" pod "6202aad3-67d4-4cec-8cbc-cdede3f9bb7c" (UID: "6202aad3-67d4-4cec-8cbc-cdede3f9bb7c"). InnerVolumeSpecName "kube-api-access-6876g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:57.915626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:57.902134    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6202aad3-67d4-4cec-8cbc-cdede3f9bb7c-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:57.915626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:57.902175    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6876g\" (UniqueName: \"kubernetes.io/projected/6202aad3-67d4-4cec-8cbc-cdede3f9bb7c-kube-api-access-6876g\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:57.993666 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:57.986200    4421 generic.go:356] "Generic (PLEG): container finished" podID="f91c8aeb-d184-4b0c-bd49-7297d51cf271" containerID="910c741a7b67da55e14e1c32fd9afa0bb68aa9030872c5f189ce92e17f531316" exitCode=0
Nov 03 20:33:57.993666 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:57.986310    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9"
Nov 03 20:33:57.993666 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:57.986348    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9" event={"ID":"f91c8aeb-d184-4b0c-bd49-7297d51cf271","Type":"ContainerDied","Data":"910c741a7b67da55e14e1c32fd9afa0bb68aa9030872c5f189ce92e17f531316"}
Nov 03 20:33:57.993666 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:57.986454    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9" event={"ID":"f91c8aeb-d184-4b0c-bd49-7297d51cf271","Type":"ContainerDied","Data":"3f8eaf63910b47984fe405bab3fae751cee0e9c369c380c81480029be21d2e04"}
Nov 03 20:33:57.993666 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:57.986487    4421 scope.go:122] "RemoveContainer" containerID="910c741a7b67da55e14e1c32fd9afa0bb68aa9030872c5f189ce92e17f531316"
Nov 03 20:33:57.997648 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:57.994469    4421 generic.go:356] "Generic (PLEG): container finished" podID="6202aad3-67d4-4cec-8cbc-cdede3f9bb7c" containerID="b03e0696a9d1536263b4dd757936cb855706273e952a478b498bcbf8722974d3" exitCode=0
Nov 03 20:33:57.997648 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:57.994627    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg" event={"ID":"6202aad3-67d4-4cec-8cbc-cdede3f9bb7c","Type":"ContainerDied","Data":"b03e0696a9d1536263b4dd757936cb855706273e952a478b498bcbf8722974d3"}
Nov 03 20:33:57.997648 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:57.994697    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg" event={"ID":"6202aad3-67d4-4cec-8cbc-cdede3f9bb7c","Type":"ContainerDied","Data":"6fd35673098bab0273c3fed960d048c9393fd5f906d0365695f1243a28602b31"}
Nov 03 20:33:57.997648 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:57.994803    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg"
Nov 03 20:33:58.012210 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:58.012189    4421 scope.go:122] "RemoveContainer" containerID="910c741a7b67da55e14e1c32fd9afa0bb68aa9030872c5f189ce92e17f531316"
Nov 03 20:33:58.012737 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:33:58.012704    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"910c741a7b67da55e14e1c32fd9afa0bb68aa9030872c5f189ce92e17f531316\": container with ID starting with 910c741a7b67da55e14e1c32fd9afa0bb68aa9030872c5f189ce92e17f531316 not found: ID does not exist" containerID="910c741a7b67da55e14e1c32fd9afa0bb68aa9030872c5f189ce92e17f531316"
Nov 03 20:33:58.012836 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:58.012745    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"910c741a7b67da55e14e1c32fd9afa0bb68aa9030872c5f189ce92e17f531316"} err="failed to get container status \"910c741a7b67da55e14e1c32fd9afa0bb68aa9030872c5f189ce92e17f531316\": rpc error: code = NotFound desc = could not find container \"910c741a7b67da55e14e1c32fd9afa0bb68aa9030872c5f189ce92e17f531316\": container with ID starting with 910c741a7b67da55e14e1c32fd9afa0bb68aa9030872c5f189ce92e17f531316 not found: ID does not exist"
Nov 03 20:33:58.012836 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:58.012767    4421 scope.go:122] "RemoveContainer" containerID="b03e0696a9d1536263b4dd757936cb855706273e952a478b498bcbf8722974d3"
Nov 03 20:33:58.025199 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:58.025179    4421 scope.go:122] "RemoveContainer" containerID="b03e0696a9d1536263b4dd757936cb855706273e952a478b498bcbf8722974d3"
Nov 03 20:33:58.025476 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:33:58.025458    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b03e0696a9d1536263b4dd757936cb855706273e952a478b498bcbf8722974d3\": container with ID starting with b03e0696a9d1536263b4dd757936cb855706273e952a478b498bcbf8722974d3 not found: ID does not exist" containerID="b03e0696a9d1536263b4dd757936cb855706273e952a478b498bcbf8722974d3"
Nov 03 20:33:58.025545 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:58.025487    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03e0696a9d1536263b4dd757936cb855706273e952a478b498bcbf8722974d3"} err="failed to get container status \"b03e0696a9d1536263b4dd757936cb855706273e952a478b498bcbf8722974d3\": rpc error: code = NotFound desc = could not find container \"b03e0696a9d1536263b4dd757936cb855706273e952a478b498bcbf8722974d3\": container with ID starting with b03e0696a9d1536263b4dd757936cb855706273e952a478b498bcbf8722974d3 not found: ID does not exist"
Nov 03 20:33:58.028467 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:58.028442    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9"]
Nov 03 20:33:58.033106 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:58.033083    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9fbc61-6c795d7dc9-c6zd9"]
Nov 03 20:33:58.043261 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:58.043190    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg"]
Nov 03 20:33:58.053657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:33:58.051213    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9fbc61-5c7c56cb6-74fcg"]
Nov 03 20:34:00.022929 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:00.022889    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6202aad3-67d4-4cec-8cbc-cdede3f9bb7c" path="/var/lib/kubelet/pods/6202aad3-67d4-4cec-8cbc-cdede3f9bb7c/volumes"
Nov 03 20:34:00.023681 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:00.023659    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f91c8aeb-d184-4b0c-bd49-7297d51cf271" path="/var/lib/kubelet/pods/f91c8aeb-d184-4b0c-bd49-7297d51cf271/volumes"
Nov 03 20:34:09.628177 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:09.628145    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2"]
Nov 03 20:34:09.628831 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:09.628414    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2" podUID="4524e977-ff54-4424-bcdb-28764428b1ef" containerName="maas-api" containerID="cri-o://d44beb1a938ffffbcaf9c9eea9b9b7e93ead4fba26724187233f30f429641886" gracePeriod=30
Nov 03 20:34:10.224195 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:10.224160    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2"
Nov 03 20:34:10.230238 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:10.230208    4421 generic.go:356] "Generic (PLEG): container finished" podID="4524e977-ff54-4424-bcdb-28764428b1ef" containerID="d44beb1a938ffffbcaf9c9eea9b9b7e93ead4fba26724187233f30f429641886" exitCode=0
Nov 03 20:34:10.230379 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:10.230247    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2"
Nov 03 20:34:10.230379 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:10.230255    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2" event={"ID":"4524e977-ff54-4424-bcdb-28764428b1ef","Type":"ContainerDied","Data":"d44beb1a938ffffbcaf9c9eea9b9b7e93ead4fba26724187233f30f429641886"}
Nov 03 20:34:10.230379 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:10.230323    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2" event={"ID":"4524e977-ff54-4424-bcdb-28764428b1ef","Type":"ContainerDied","Data":"5f2eade1e0c99e2c904a8ff7b24d07ba3a5509ee6eed6ad0513c12bb6515942d"}
Nov 03 20:34:10.230379 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:10.230354    4421 scope.go:122] "RemoveContainer" containerID="d44beb1a938ffffbcaf9c9eea9b9b7e93ead4fba26724187233f30f429641886"
Nov 03 20:34:10.242762 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:10.242741    4421 scope.go:122] "RemoveContainer" containerID="d44beb1a938ffffbcaf9c9eea9b9b7e93ead4fba26724187233f30f429641886"
Nov 03 20:34:10.243078 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:34:10.243052    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d44beb1a938ffffbcaf9c9eea9b9b7e93ead4fba26724187233f30f429641886\": container with ID starting with d44beb1a938ffffbcaf9c9eea9b9b7e93ead4fba26724187233f30f429641886 not found: ID does not exist" containerID="d44beb1a938ffffbcaf9c9eea9b9b7e93ead4fba26724187233f30f429641886"
Nov 03 20:34:10.243204 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:10.243088    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d44beb1a938ffffbcaf9c9eea9b9b7e93ead4fba26724187233f30f429641886"} err="failed to get container status \"d44beb1a938ffffbcaf9c9eea9b9b7e93ead4fba26724187233f30f429641886\": rpc error: code = NotFound desc = could not find container \"d44beb1a938ffffbcaf9c9eea9b9b7e93ead4fba26724187233f30f429641886\": container with ID starting with d44beb1a938ffffbcaf9c9eea9b9b7e93ead4fba26724187233f30f429641886 not found: ID does not exist"
Nov 03 20:34:10.286852 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:10.286825    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4524e977-ff54-4424-bcdb-28764428b1ef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4524e977-ff54-4424-bcdb-28764428b1ef-maas-api-tls\") pod \"4524e977-ff54-4424-bcdb-28764428b1ef\" (UID: \"4524e977-ff54-4424-bcdb-28764428b1ef\") "
Nov 03 20:34:10.287062 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:10.286935    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4524e977-ff54-4424-bcdb-28764428b1ef-kube-api-access-jrcsd\" (UniqueName: \"kubernetes.io/projected/4524e977-ff54-4424-bcdb-28764428b1ef-kube-api-access-jrcsd\") pod \"4524e977-ff54-4424-bcdb-28764428b1ef\" (UID: \"4524e977-ff54-4424-bcdb-28764428b1ef\") "
Nov 03 20:34:10.287150 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:10.287116    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4524e977-ff54-4424-bcdb-28764428b1ef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4524e977-ff54-4424-bcdb-28764428b1ef-openshift-service-ca\") pod \"4524e977-ff54-4424-bcdb-28764428b1ef\" (UID: \"4524e977-ff54-4424-bcdb-28764428b1ef\") "
Nov 03 20:34:10.287217 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:10.287171    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4524e977-ff54-4424-bcdb-28764428b1ef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4524e977-ff54-4424-bcdb-28764428b1ef-metrics-certs\") pod \"4524e977-ff54-4424-bcdb-28764428b1ef\" (UID: \"4524e977-ff54-4424-bcdb-28764428b1ef\") "
Nov 03 20:34:10.287646 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:10.287589    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4524e977-ff54-4424-bcdb-28764428b1ef-openshift-service-ca" pod "4524e977-ff54-4424-bcdb-28764428b1ef" (UID: "4524e977-ff54-4424-bcdb-28764428b1ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:34:10.288114 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:10.288089    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4524e977-ff54-4424-bcdb-28764428b1ef-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:34:10.290417 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:10.290383    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4524e977-ff54-4424-bcdb-28764428b1ef-maas-api-tls" pod "4524e977-ff54-4424-bcdb-28764428b1ef" (UID: "4524e977-ff54-4424-bcdb-28764428b1ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:34:10.290715 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:10.290694    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4524e977-ff54-4424-bcdb-28764428b1ef-metrics-certs" pod "4524e977-ff54-4424-bcdb-28764428b1ef" (UID: "4524e977-ff54-4424-bcdb-28764428b1ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:34:10.293533 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:10.293505    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4524e977-ff54-4424-bcdb-28764428b1ef-kube-api-access-jrcsd" pod "4524e977-ff54-4424-bcdb-28764428b1ef" (UID: "4524e977-ff54-4424-bcdb-28764428b1ef"). InnerVolumeSpecName "kube-api-access-jrcsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:10.393674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:10.393291    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4524e977-ff54-4424-bcdb-28764428b1ef-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:34:10.393674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:10.393332    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrcsd\" (UniqueName: \"kubernetes.io/projected/4524e977-ff54-4424-bcdb-28764428b1ef-kube-api-access-jrcsd\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:34:10.393674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:10.393347    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4524e977-ff54-4424-bcdb-28764428b1ef-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:34:10.565456 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:10.565416    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2"]
Nov 03 20:34:10.570559 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:10.570526    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9fbc61-85779d5ddc-nstw2"]
Nov 03 20:34:12.018442 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.018410    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4524e977-ff54-4424-bcdb-28764428b1ef" path="/var/lib/kubelet/pods/4524e977-ff54-4424-bcdb-28764428b1ef/volumes"
Nov 03 20:34:12.500968 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.500935    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d"]
Nov 03 20:34:12.502524 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.502285    4421 cpu_manager.go:414] "containerMap: removing container" podUID="6202aad3-67d4-4cec-8cbc-cdede3f9bb7c" containerName="payload-processing"
Nov 03 20:34:12.502524 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.502313    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6202aad3-67d4-4cec-8cbc-cdede3f9bb7c" containerName="payload-processing"
Nov 03 20:34:12.502524 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.502345    4421 cpu_manager.go:414] "containerMap: removing container" podUID="68f60111-d601-492f-b82e-9db18a818e3a" containerName="payload-pre-processing"
Nov 03 20:34:12.502524 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.502354    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68f60111-d601-492f-b82e-9db18a818e3a" containerName="payload-pre-processing"
Nov 03 20:34:12.502524 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.502377    4421 cpu_manager.go:414] "containerMap: removing container" podUID="f91c8aeb-d184-4b0c-bd49-7297d51cf271" containerName="payload-pre-processing"
Nov 03 20:34:12.502524 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.502385    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f91c8aeb-d184-4b0c-bd49-7297d51cf271" containerName="payload-pre-processing"
Nov 03 20:34:12.502524 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.502402    4421 cpu_manager.go:414] "containerMap: removing container" podUID="4524e977-ff54-4424-bcdb-28764428b1ef" containerName="maas-api"
Nov 03 20:34:12.502524 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.502409    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4524e977-ff54-4424-bcdb-28764428b1ef" containerName="maas-api"
Nov 03 20:34:12.502524 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.502435    4421 cpu_manager.go:414] "containerMap: removing container" podUID="a020faaa-cf90-466e-8715-ccf5f987b158" containerName="payload-processing"
Nov 03 20:34:12.502524 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.502444    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a020faaa-cf90-466e-8715-ccf5f987b158" containerName="payload-processing"
Nov 03 20:34:12.503182 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.502694    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6202aad3-67d4-4cec-8cbc-cdede3f9bb7c" containerName="payload-processing"
Nov 03 20:34:12.503182 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.502716    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="68f60111-d601-492f-b82e-9db18a818e3a" containerName="payload-pre-processing"
Nov 03 20:34:12.503182 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.502733    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f91c8aeb-d184-4b0c-bd49-7297d51cf271" containerName="payload-pre-processing"
Nov 03 20:34:12.503182 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.502743    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="4524e977-ff54-4424-bcdb-28764428b1ef" containerName="maas-api"
Nov 03 20:34:12.503182 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.502755    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a020faaa-cf90-466e-8715-ccf5f987b158" containerName="payload-processing"
Nov 03 20:34:12.508942 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.508918    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d"
Nov 03 20:34:12.513940 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.513916    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-03c633-serving-cert\""
Nov 03 20:34:12.516493 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.516469    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d"]
Nov 03 20:34:12.632683 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.632549    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c1a8f22-edce-40d8-8216-adb79f5cd692-metrics-certs\") pod \"maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d\" (UID: \"9c1a8f22-edce-40d8-8216-adb79f5cd692\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d"
Nov 03 20:34:12.632864 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.632713    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c1a8f22-edce-40d8-8216-adb79f5cd692-maas-api-tls\") pod \"maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d\" (UID: \"9c1a8f22-edce-40d8-8216-adb79f5cd692\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d"
Nov 03 20:34:12.632864 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.632849    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1a8f22-edce-40d8-8216-adb79f5cd692-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d\" (UID: \"9c1a8f22-edce-40d8-8216-adb79f5cd692\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d"
Nov 03 20:34:12.632995 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.632914    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xhhc\" (UniqueName: \"kubernetes.io/projected/9c1a8f22-edce-40d8-8216-adb79f5cd692-kube-api-access-2xhhc\") pod \"maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d\" (UID: \"9c1a8f22-edce-40d8-8216-adb79f5cd692\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d"
Nov 03 20:34:12.740633 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.734694    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1a8f22-edce-40d8-8216-adb79f5cd692-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d\" (UID: \"9c1a8f22-edce-40d8-8216-adb79f5cd692\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d"
Nov 03 20:34:12.740633 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.734784    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xhhc\" (UniqueName: \"kubernetes.io/projected/9c1a8f22-edce-40d8-8216-adb79f5cd692-kube-api-access-2xhhc\") pod \"maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d\" (UID: \"9c1a8f22-edce-40d8-8216-adb79f5cd692\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d"
Nov 03 20:34:12.740633 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.735115    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c1a8f22-edce-40d8-8216-adb79f5cd692-metrics-certs\") pod \"maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d\" (UID: \"9c1a8f22-edce-40d8-8216-adb79f5cd692\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d"
Nov 03 20:34:12.740633 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.735184    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c1a8f22-edce-40d8-8216-adb79f5cd692-maas-api-tls\") pod \"maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d\" (UID: \"9c1a8f22-edce-40d8-8216-adb79f5cd692\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d"
Nov 03 20:34:12.740633 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.736503    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1a8f22-edce-40d8-8216-adb79f5cd692-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d\" (UID: \"9c1a8f22-edce-40d8-8216-adb79f5cd692\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d"
Nov 03 20:34:12.773424 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.767533    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xhhc\" (UniqueName: \"kubernetes.io/projected/9c1a8f22-edce-40d8-8216-adb79f5cd692-kube-api-access-2xhhc\") pod \"maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d\" (UID: \"9c1a8f22-edce-40d8-8216-adb79f5cd692\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d"
Nov 03 20:34:12.773424 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.768037    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c1a8f22-edce-40d8-8216-adb79f5cd692-metrics-certs\") pod \"maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d\" (UID: \"9c1a8f22-edce-40d8-8216-adb79f5cd692\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d"
Nov 03 20:34:12.784630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.776621    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c1a8f22-edce-40d8-8216-adb79f5cd692-maas-api-tls\") pod \"maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d\" (UID: \"9c1a8f22-edce-40d8-8216-adb79f5cd692\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d"
Nov 03 20:34:12.828560 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:12.828211    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d"
Nov 03 20:34:13.202138 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:13.202095    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d"]
Nov 03 20:34:13.204738 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:34:13.204688    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1a8f22_edce_40d8_8216_adb79f5cd692.slice/crio-b34c0ff64732f9bbbee617330cfae06e0daf7f3d3d3e7741b6dbd9421e504150 WatchSource:0}: Error finding container b34c0ff64732f9bbbee617330cfae06e0daf7f3d3d3e7741b6dbd9421e504150: Status 404 returned error can't find the container with id b34c0ff64732f9bbbee617330cfae06e0daf7f3d3d3e7741b6dbd9421e504150
Nov 03 20:34:13.259064 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:13.258639    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d" event={"ID":"9c1a8f22-edce-40d8-8216-adb79f5cd692","Type":"ContainerStarted","Data":"b34c0ff64732f9bbbee617330cfae06e0daf7f3d3d3e7741b6dbd9421e504150"}
Nov 03 20:34:14.304473 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:14.304422    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d" event={"ID":"9c1a8f22-edce-40d8-8216-adb79f5cd692","Type":"ContainerStarted","Data":"599deb32f0dc7c836d8a6749bb973c74e8e4d6fbed20a55a4f9f9d23bb11e7a9"}
Nov 03 20:34:14.305034 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:14.304751    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d"
Nov 03 20:34:14.328392 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:14.328333    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d" podStartSLOduration=1.7972681590000001 podStartE2EDuration="2.328315392s" totalImagesPullingTime="531.047233ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:14.325796359 +0000 UTC m=+1984.458924820" watchObservedRunningTime="2026-11-03 20:34:14.328315392 +0000 UTC m=+1984.461443856"
Nov 03 20:34:17.314225 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:17.314187    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"]
Nov 03 20:34:17.314688 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:17.314424    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr" podUID="4fbddfde-9453-458d-b50b-05a3deeabf74" containerName="istio-proxy" containerID="cri-o://4c788c30b6bfa3f8c05cc5d936aadd6b9b35a439722bb8d5a71bf38b7a2fc82d" gracePeriod=30
Nov 03 20:34:19.630797 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:34:19.630759    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:34:20.324032 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:20.324001    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d"
Nov 03 20:34:22.447995 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.447956    4421 generic.go:356] "Generic (PLEG): container finished" podID="4fbddfde-9453-458d-b50b-05a3deeabf74" containerID="4c788c30b6bfa3f8c05cc5d936aadd6b9b35a439722bb8d5a71bf38b7a2fc82d" exitCode=0
Nov 03 20:34:22.448380 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.448010    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr" event={"ID":"4fbddfde-9453-458d-b50b-05a3deeabf74","Type":"ContainerDied","Data":"4c788c30b6bfa3f8c05cc5d936aadd6b9b35a439722bb8d5a71bf38b7a2fc82d"}
Nov 03 20:34:22.448380 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.448091    4421 scope.go:122] "RemoveContainer" containerID="805470c43d8e2f5741e50fc54f9579d00ba1df5ffcefb0db8d3d68d1197816ef"
Nov 03 20:34:22.664335 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.664261    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:34:22.824833 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.812163    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-workload-socket\") pod \"4fbddfde-9453-458d-b50b-05a3deeabf74\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") "
Nov 03 20:34:22.825160 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.825130    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-podinfo\") pod \"4fbddfde-9453-458d-b50b-05a3deeabf74\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") "
Nov 03 20:34:22.825931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.825876    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-credential-socket\") pod \"4fbddfde-9453-458d-b50b-05a3deeabf74\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") "
Nov 03 20:34:22.826447 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.826427    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-workload-certs\") pod \"4fbddfde-9453-458d-b50b-05a3deeabf74\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") "
Nov 03 20:34:22.827155 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.827136    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-token\" (UniqueName: \"kubernetes.io/projected/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-token\") pod \"4fbddfde-9453-458d-b50b-05a3deeabf74\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") "
Nov 03 20:34:22.827734 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.827714    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-envoy\") pod \"4fbddfde-9453-458d-b50b-05a3deeabf74\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") "
Nov 03 20:34:22.828426 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.828409    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fbddfde-9453-458d-b50b-05a3deeabf74-kube-api-access-g9d6m\" (UniqueName: \"kubernetes.io/projected/4fbddfde-9453-458d-b50b-05a3deeabf74-kube-api-access-g9d6m\") pod \"4fbddfde-9453-458d-b50b-05a3deeabf74\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") "
Nov 03 20:34:22.838151 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.828988    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-data\") pod \"4fbddfde-9453-458d-b50b-05a3deeabf74\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") "
Nov 03 20:34:22.843008 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.842981    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fbddfde-9453-458d-b50b-05a3deeabf74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fbddfde-9453-458d-b50b-05a3deeabf74-istiod-ca-cert\") pod \"4fbddfde-9453-458d-b50b-05a3deeabf74\" (UID: \"4fbddfde-9453-458d-b50b-05a3deeabf74\") "
Nov 03 20:34:22.848573 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.822503    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-workload-socket" pod "4fbddfde-9453-458d-b50b-05a3deeabf74" (UID: "4fbddfde-9453-458d-b50b-05a3deeabf74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:22.848573 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.826361    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-credential-socket" pod "4fbddfde-9453-458d-b50b-05a3deeabf74" (UID: "4fbddfde-9453-458d-b50b-05a3deeabf74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:22.848573 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.828358    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-workload-certs" pod "4fbddfde-9453-458d-b50b-05a3deeabf74" (UID: "4fbddfde-9453-458d-b50b-05a3deeabf74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:22.848573 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.842742    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-data" pod "4fbddfde-9453-458d-b50b-05a3deeabf74" (UID: "4fbddfde-9453-458d-b50b-05a3deeabf74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:22.848573 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.846543    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fbddfde-9453-458d-b50b-05a3deeabf74-istiod-ca-cert" pod "4fbddfde-9453-458d-b50b-05a3deeabf74" (UID: "4fbddfde-9453-458d-b50b-05a3deeabf74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:34:22.887049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.875641    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-envoy" pod "4fbddfde-9453-458d-b50b-05a3deeabf74" (UID: "4fbddfde-9453-458d-b50b-05a3deeabf74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:34:22.887049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.875835    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-token" pod "4fbddfde-9453-458d-b50b-05a3deeabf74" (UID: "4fbddfde-9453-458d-b50b-05a3deeabf74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:22.897672 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.895234    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fbddfde-9453-458d-b50b-05a3deeabf74-kube-api-access-g9d6m" pod "4fbddfde-9453-458d-b50b-05a3deeabf74" (UID: "4fbddfde-9453-458d-b50b-05a3deeabf74"). InnerVolumeSpecName "kube-api-access-g9d6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:34:22.898704 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.898579    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-podinfo" pod "4fbddfde-9453-458d-b50b-05a3deeabf74" (UID: "4fbddfde-9453-458d-b50b-05a3deeabf74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:34:22.950288 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.950252    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:34:22.950288 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.950285    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:34:22.950288 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.950294    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:34:22.950667 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.950305    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9d6m\" (UniqueName: \"kubernetes.io/projected/4fbddfde-9453-458d-b50b-05a3deeabf74-kube-api-access-g9d6m\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:34:22.950667 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.950315    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:34:22.950667 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.950323    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fbddfde-9453-458d-b50b-05a3deeabf74-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:34:22.950667 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.950332    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:34:22.950667 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.950340    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fbddfde-9453-458d-b50b-05a3deeabf74-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:34:22.950667 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:22.950348    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fbddfde-9453-458d-b50b-05a3deeabf74-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:34:23.457319 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:23.457287    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"
Nov 03 20:34:23.457319 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:23.457306    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr" event={"ID":"4fbddfde-9453-458d-b50b-05a3deeabf74","Type":"ContainerDied","Data":"ec92531b3473cf3bc6eb1d590e1785eccce8049ee508cae31df5a85abc7da470"}
Nov 03 20:34:23.457833 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:23.457365    4421 scope.go:122] "RemoveContainer" containerID="4c788c30b6bfa3f8c05cc5d936aadd6b9b35a439722bb8d5a71bf38b7a2fc82d"
Nov 03 20:34:23.492143 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:23.492116    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"]
Nov 03 20:34:23.500866 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:23.500842    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-65cfr"]
Nov 03 20:34:24.018779 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.018748    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fbddfde-9453-458d-b50b-05a3deeabf74" path="/var/lib/kubelet/pods/4fbddfde-9453-458d-b50b-05a3deeabf74/volumes"
Nov 03 20:34:24.611214 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.611185    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw"]
Nov 03 20:34:24.612033 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.612018    4421 cpu_manager.go:414] "containerMap: removing container" podUID="4fbddfde-9453-458d-b50b-05a3deeabf74" containerName="istio-proxy"
Nov 03 20:34:24.612093 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.612034    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fbddfde-9453-458d-b50b-05a3deeabf74" containerName="istio-proxy"
Nov 03 20:34:24.612093 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.612060    4421 cpu_manager.go:414] "containerMap: removing container" podUID="4fbddfde-9453-458d-b50b-05a3deeabf74" containerName="istio-proxy"
Nov 03 20:34:24.612093 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.612065    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fbddfde-9453-458d-b50b-05a3deeabf74" containerName="istio-proxy"
Nov 03 20:34:24.612200 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.612191    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fbddfde-9453-458d-b50b-05a3deeabf74" containerName="istio-proxy"
Nov 03 20:34:24.617552 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.617527    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw"
Nov 03 20:34:24.624064 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.624039    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-14ebbb\""
Nov 03 20:34:24.627702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.627647    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-14ebbb-dockercfg-q9t9n\""
Nov 03 20:34:24.629498 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.629473    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw"]
Nov 03 20:34:24.636838 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.636811    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq"]
Nov 03 20:34:24.638402 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.638382    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fbddfde-9453-458d-b50b-05a3deeabf74" containerName="istio-proxy"
Nov 03 20:34:24.643071 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.643049    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq"
Nov 03 20:34:24.662087 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.662052    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq"]
Nov 03 20:34:24.681386 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.681360    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae207be-6a8f-4773-bc31-527edb88be5e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw\" (UID: \"0ae207be-6a8f-4773-bc31-527edb88be5e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw"
Nov 03 20:34:24.681537 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.681422    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a0dff8b-05ed-4261-a30c-08ced3980a2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq\" (UID: \"3a0dff8b-05ed-4261-a30c-08ced3980a2c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq"
Nov 03 20:34:24.681657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.681541    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5hqg\" (UniqueName: \"kubernetes.io/projected/0ae207be-6a8f-4773-bc31-527edb88be5e-kube-api-access-c5hqg\") pod \"payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw\" (UID: \"0ae207be-6a8f-4773-bc31-527edb88be5e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw"
Nov 03 20:34:24.681784 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.681764    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbgbj\" (UniqueName: \"kubernetes.io/projected/3a0dff8b-05ed-4261-a30c-08ced3980a2c-kube-api-access-sbgbj\") pod \"payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq\" (UID: \"3a0dff8b-05ed-4261-a30c-08ced3980a2c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq"
Nov 03 20:34:24.785621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.784767    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a0dff8b-05ed-4261-a30c-08ced3980a2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq\" (UID: \"3a0dff8b-05ed-4261-a30c-08ced3980a2c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq"
Nov 03 20:34:24.785621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.784870    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5hqg\" (UniqueName: \"kubernetes.io/projected/0ae207be-6a8f-4773-bc31-527edb88be5e-kube-api-access-c5hqg\") pod \"payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw\" (UID: \"0ae207be-6a8f-4773-bc31-527edb88be5e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw"
Nov 03 20:34:24.785621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.785018    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbgbj\" (UniqueName: \"kubernetes.io/projected/3a0dff8b-05ed-4261-a30c-08ced3980a2c-kube-api-access-sbgbj\") pod \"payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq\" (UID: \"3a0dff8b-05ed-4261-a30c-08ced3980a2c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq"
Nov 03 20:34:24.785621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.785266    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae207be-6a8f-4773-bc31-527edb88be5e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw\" (UID: \"0ae207be-6a8f-4773-bc31-527edb88be5e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw"
Nov 03 20:34:24.789620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.786225    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae207be-6a8f-4773-bc31-527edb88be5e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw\" (UID: \"0ae207be-6a8f-4773-bc31-527edb88be5e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw"
Nov 03 20:34:24.789620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.787043    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a0dff8b-05ed-4261-a30c-08ced3980a2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq\" (UID: \"3a0dff8b-05ed-4261-a30c-08ced3980a2c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq"
Nov 03 20:34:24.832627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.825064    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbgbj\" (UniqueName: \"kubernetes.io/projected/3a0dff8b-05ed-4261-a30c-08ced3980a2c-kube-api-access-sbgbj\") pod \"payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq\" (UID: \"3a0dff8b-05ed-4261-a30c-08ced3980a2c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq"
Nov 03 20:34:24.832627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.826332    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5hqg\" (UniqueName: \"kubernetes.io/projected/0ae207be-6a8f-4773-bc31-527edb88be5e-kube-api-access-c5hqg\") pod \"payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw\" (UID: \"0ae207be-6a8f-4773-bc31-527edb88be5e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw"
Nov 03 20:34:24.948618 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.946069    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw"
Nov 03 20:34:24.991618 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:24.986954    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq"
Nov 03 20:34:25.414455 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:34:25.414417    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0dff8b_05ed_4261_a30c_08ced3980a2c.slice/crio-3211fc8c13df7f8e0cac28cc1b8a9df17c1ea1559c9f74060ccbc3c7e7a0e2b6 WatchSource:0}: Error finding container 3211fc8c13df7f8e0cac28cc1b8a9df17c1ea1559c9f74060ccbc3c7e7a0e2b6: Status 404 returned error can't find the container with id 3211fc8c13df7f8e0cac28cc1b8a9df17c1ea1559c9f74060ccbc3c7e7a0e2b6
Nov 03 20:34:25.415639 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:25.415615    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq"]
Nov 03 20:34:25.415906 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:34:25.415766    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae207be_6a8f_4773_bc31_527edb88be5e.slice/crio-f3e88cec57f935da13825fe505c78be7203932dfd221a48ad73448824d5cfd41 WatchSource:0}: Error finding container f3e88cec57f935da13825fe505c78be7203932dfd221a48ad73448824d5cfd41: Status 404 returned error can't find the container with id f3e88cec57f935da13825fe505c78be7203932dfd221a48ad73448824d5cfd41
Nov 03 20:34:25.420668 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:25.420635    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw"]
Nov 03 20:34:25.477851 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:25.477827    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw" event={"ID":"0ae207be-6a8f-4773-bc31-527edb88be5e","Type":"ContainerStarted","Data":"f3e88cec57f935da13825fe505c78be7203932dfd221a48ad73448824d5cfd41"}
Nov 03 20:34:25.479352 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:25.479324    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq" event={"ID":"3a0dff8b-05ed-4261-a30c-08ced3980a2c","Type":"ContainerStarted","Data":"3211fc8c13df7f8e0cac28cc1b8a9df17c1ea1559c9f74060ccbc3c7e7a0e2b6"}
Nov 03 20:34:26.488966 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:26.488931    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq" event={"ID":"3a0dff8b-05ed-4261-a30c-08ced3980a2c","Type":"ContainerStarted","Data":"ea0e38b5dc46921faeac45d30eeee38c55748ba3462c9a77155a331e8c5d2b80"}
Nov 03 20:34:26.489451 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:26.489022    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq"
Nov 03 20:34:26.490660 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:26.490638    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw" event={"ID":"0ae207be-6a8f-4773-bc31-527edb88be5e","Type":"ContainerStarted","Data":"290d215e8ff95668de33fe2bf943b9b7bb56fcf991aaaae62de907aa9d462f59"}
Nov 03 20:34:26.490760 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:26.490690    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw"
Nov 03 20:34:26.510960 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:26.510912    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq" podStartSLOduration=2.510898882 podStartE2EDuration="2.510898882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:26.508634796 +0000 UTC m=+1996.641763253" watchObservedRunningTime="2026-11-03 20:34:26.510898882 +0000 UTC m=+1996.644027340"
Nov 03 20:34:26.540630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:26.540553    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw" podStartSLOduration=2.540538794 podStartE2EDuration="2.540538794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:26.535736103 +0000 UTC m=+1996.668864566" watchObservedRunningTime="2026-11-03 20:34:26.540538794 +0000 UTC m=+1996.673667283"
Nov 03 20:34:37.536440 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:37.536408    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw"
Nov 03 20:34:37.551833 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:37.551800    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq"
Nov 03 20:34:49.631152 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:34:49.631074    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:34:51.625055 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:51.625022    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m"]
Nov 03 20:34:51.631502 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:51.631477    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m"
Nov 03 20:34:51.638186 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:51.638158    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m"]
Nov 03 20:34:51.639925 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:51.639904    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-c7aecb\""
Nov 03 20:34:51.718648 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:51.714157    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-c7aecb-dockercfg-jq9j5\""
Nov 03 20:34:51.788640 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:51.739437    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhpj6\" (UniqueName: \"kubernetes.io/projected/5b495a05-8e6f-4b43-b9d3-20ad6b499731-kube-api-access-jhpj6\") pod \"payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m\" (UID: \"5b495a05-8e6f-4b43-b9d3-20ad6b499731\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m"
Nov 03 20:34:51.788640 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:51.739534    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b495a05-8e6f-4b43-b9d3-20ad6b499731-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m\" (UID: \"5b495a05-8e6f-4b43-b9d3-20ad6b499731\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m"
Nov 03 20:34:51.907117 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:51.907013    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhpj6\" (UniqueName: \"kubernetes.io/projected/5b495a05-8e6f-4b43-b9d3-20ad6b499731-kube-api-access-jhpj6\") pod \"payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m\" (UID: \"5b495a05-8e6f-4b43-b9d3-20ad6b499731\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m"
Nov 03 20:34:51.907438 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:51.907412    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b495a05-8e6f-4b43-b9d3-20ad6b499731-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m\" (UID: \"5b495a05-8e6f-4b43-b9d3-20ad6b499731\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m"
Nov 03 20:34:51.908927 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:51.908876    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b495a05-8e6f-4b43-b9d3-20ad6b499731-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m\" (UID: \"5b495a05-8e6f-4b43-b9d3-20ad6b499731\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m"
Nov 03 20:34:52.021620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:52.020772    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhpj6\" (UniqueName: \"kubernetes.io/projected/5b495a05-8e6f-4b43-b9d3-20ad6b499731-kube-api-access-jhpj6\") pod \"payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m\" (UID: \"5b495a05-8e6f-4b43-b9d3-20ad6b499731\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m"
Nov 03 20:34:52.258042 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:52.258013    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m"
Nov 03 20:34:52.549054 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:52.549023    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m"]
Nov 03 20:34:52.549231 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:34:52.549180    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b495a05_8e6f_4b43_b9d3_20ad6b499731.slice/crio-b240b885cca00d87ef5428e02504df2623bb657c39377294d0366c040c15533e WatchSource:0}: Error finding container b240b885cca00d87ef5428e02504df2623bb657c39377294d0366c040c15533e: Status 404 returned error can't find the container with id b240b885cca00d87ef5428e02504df2623bb657c39377294d0366c040c15533e
Nov 03 20:34:53.011278 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:53.011242    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m" event={"ID":"5b495a05-8e6f-4b43-b9d3-20ad6b499731","Type":"ContainerStarted","Data":"0cab1cf353d3092be16a0075f6a6582051379fedad8f3b5139e7f3ef0a3c043f"}
Nov 03 20:34:53.011717 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:53.011297    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m" event={"ID":"5b495a05-8e6f-4b43-b9d3-20ad6b499731","Type":"ContainerStarted","Data":"b240b885cca00d87ef5428e02504df2623bb657c39377294d0366c040c15533e"}
Nov 03 20:34:53.011717 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:53.011433    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m"
Nov 03 20:34:53.039507 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:34:53.039446    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m" podStartSLOduration=2.03942965 podStartE2EDuration="2.03942965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:53.034161142 +0000 UTC m=+2023.167289604" watchObservedRunningTime="2026-11-03 20:34:53.03942965 +0000 UTC m=+2023.172558113"
Nov 03 20:35:04.030976 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:04.030943    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m"
Nov 03 20:35:05.951626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:05.945658    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"]
Nov 03 20:35:05.990691 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:05.989928    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:05.995662 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:05.995633    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-6350b9-openshift-default-dockercfg-hsb7k\""
Nov 03 20:35:06.175623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.170680    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-workload-socket\") pod \"e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") " pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.175623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.170764    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-workload-certs\") pod \"e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") " pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.175623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.170797    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwkbd\" (UniqueName: \"kubernetes.io/projected/56233474-fdc7-4bcb-aa2a-db87ac074168-kube-api-access-mwkbd\") pod \"e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") " pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.175623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.170960    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-podinfo\") pod \"e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") " pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.175623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.171038    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-token\") pod \"e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") " pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.175623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.171167    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-data\") pod \"e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") " pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.175623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.171242    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-envoy\") pod \"e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") " pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.175623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.171276    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56233474-fdc7-4bcb-aa2a-db87ac074168-istiod-ca-cert\") pod \"e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") " pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.175623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.171470    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-credential-socket\") pod \"e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") " pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.247159 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.246975    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"]
Nov 03 20:35:06.282422 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.279790    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-data\") pod \"e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") " pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.282422 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.279877    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-envoy\") pod \"e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") " pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.282422 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.279908    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56233474-fdc7-4bcb-aa2a-db87ac074168-istiod-ca-cert\") pod \"e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") " pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.282422 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.280068    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-credential-socket\") pod \"e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") " pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.282422 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.280146    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-workload-socket\") pod \"e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") " pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.282422 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.280188    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-workload-certs\") pod \"e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") " pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.282422 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.280214    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwkbd\" (UniqueName: \"kubernetes.io/projected/56233474-fdc7-4bcb-aa2a-db87ac074168-kube-api-access-mwkbd\") pod \"e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") " pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.282422 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.280399    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-data\") pod \"e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") " pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.282422 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.280449    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-podinfo\") pod \"e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") " pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.282422 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.280529    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-token\") pod \"e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") " pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.300937 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.286339    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-envoy\") pod \"e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") " pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.300937 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.287847    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56233474-fdc7-4bcb-aa2a-db87ac074168-istiod-ca-cert\") pod \"e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") " pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.300937 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.288217    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-credential-socket\") pod \"e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") " pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.300937 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.288453    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-workload-socket\") pod \"e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") " pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.300937 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.288750    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-workload-certs\") pod \"e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") " pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.305689 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.305661    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-podinfo\") pod \"e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") " pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.320674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.319435    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwkbd\" (UniqueName: \"kubernetes.io/projected/56233474-fdc7-4bcb-aa2a-db87ac074168-kube-api-access-mwkbd\") pod \"e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") " pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.320674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.319936    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-token\") pod \"e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") " pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.448488 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.448443    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:06.770996 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:35:06.770953    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56233474_fdc7_4bcb_aa2a_db87ac074168.slice/crio-4616839d34d5a9e2ed467d578c6c2a4ca5f4d0f2678bb33ff157677997836fb0 WatchSource:0}: Error finding container 4616839d34d5a9e2ed467d578c6c2a4ca5f4d0f2678bb33ff157677997836fb0: Status 404 returned error can't find the container with id 4616839d34d5a9e2ed467d578c6c2a4ca5f4d0f2678bb33ff157677997836fb0
Nov 03 20:35:06.773769 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.773731    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"]
Nov 03 20:35:06.774091 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.774058    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:35:06.774140 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.774125    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:35:06.774181 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:06.774165    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:35:07.245653 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:07.243495    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm" event={"ID":"56233474-fdc7-4bcb-aa2a-db87ac074168","Type":"ContainerStarted","Data":"f509b2406b74172416939d17669932f60e42e4b09fab9ec1df4f36089134c444"}
Nov 03 20:35:07.245653 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:07.243558    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm" event={"ID":"56233474-fdc7-4bcb-aa2a-db87ac074168","Type":"ContainerStarted","Data":"4616839d34d5a9e2ed467d578c6c2a4ca5f4d0f2678bb33ff157677997836fb0"}
Nov 03 20:35:07.449473 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:07.449443    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:07.455211 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:07.455186    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:07.481709 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:07.481627    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm" podStartSLOduration=2.481591708 podStartE2EDuration="2.481591708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:07.286305275 +0000 UTC m=+2037.419433742" watchObservedRunningTime="2026-11-03 20:35:07.481591708 +0000 UTC m=+2037.614720170"
Nov 03 20:35:08.295742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:08.295657    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:08.302629 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:08.301968    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:35:09.180332 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:09.180290    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg"]
Nov 03 20:35:09.185815 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:09.185789    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg"
Nov 03 20:35:09.195747 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:09.195656    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"payload-processing-external-model-e2e--e92303d8-dockercfg-99lv6\""
Nov 03 20:35:09.195747 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:09.195747    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"agc-cred-1def241ab099\""
Nov 03 20:35:09.197408 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:09.197381    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg"]
Nov 03 20:35:09.199020 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:09.198878    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"kube-root-ca.crt\""
Nov 03 20:35:09.199020 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:09.198911    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"openshift-service-ca.crt\""
Nov 03 20:35:09.199020 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:09.198967    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"payload-processing-external-model-plugins-e2e-extmod-a-db5198\""
Nov 03 20:35:09.199020 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:09.199000    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"routing-overlay\""
Nov 03 20:35:09.320540 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:09.320505    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl6qc\" (UniqueName: \"kubernetes.io/projected/f7dcbaab-2006-4334-b94d-33b242aac0ef-kube-api-access-xl6qc\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg\" (UID: \"f7dcbaab-2006-4334-b94d-33b242aac0ef\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg"
Nov 03 20:35:09.320962 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:09.320778    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7dcbaab-2006-4334-b94d-33b242aac0ef-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg\" (UID: \"f7dcbaab-2006-4334-b94d-33b242aac0ef\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg"
Nov 03 20:35:09.320962 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:09.320805    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7dcbaab-2006-4334-b94d-33b242aac0ef-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg\" (UID: \"f7dcbaab-2006-4334-b94d-33b242aac0ef\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg"
Nov 03 20:35:09.320962 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:09.320830    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7dcbaab-2006-4334-b94d-33b242aac0ef-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg\" (UID: \"f7dcbaab-2006-4334-b94d-33b242aac0ef\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg"
Nov 03 20:35:09.491627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:09.490165    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl6qc\" (UniqueName: \"kubernetes.io/projected/f7dcbaab-2006-4334-b94d-33b242aac0ef-kube-api-access-xl6qc\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg\" (UID: \"f7dcbaab-2006-4334-b94d-33b242aac0ef\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg"
Nov 03 20:35:09.491627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:09.491461    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7dcbaab-2006-4334-b94d-33b242aac0ef-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg\" (UID: \"f7dcbaab-2006-4334-b94d-33b242aac0ef\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg"
Nov 03 20:35:09.491977 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:09.491784    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7dcbaab-2006-4334-b94d-33b242aac0ef-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg\" (UID: \"f7dcbaab-2006-4334-b94d-33b242aac0ef\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg"
Nov 03 20:35:09.491977 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:09.491833    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7dcbaab-2006-4334-b94d-33b242aac0ef-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg\" (UID: \"f7dcbaab-2006-4334-b94d-33b242aac0ef\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg"
Nov 03 20:35:09.502638 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:09.501325    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7dcbaab-2006-4334-b94d-33b242aac0ef-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg\" (UID: \"f7dcbaab-2006-4334-b94d-33b242aac0ef\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg"
Nov 03 20:35:09.502638 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:09.502038    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7dcbaab-2006-4334-b94d-33b242aac0ef-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg\" (UID: \"f7dcbaab-2006-4334-b94d-33b242aac0ef\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg"
Nov 03 20:35:09.570413 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:09.553228    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl6qc\" (UniqueName: \"kubernetes.io/projected/f7dcbaab-2006-4334-b94d-33b242aac0ef-kube-api-access-xl6qc\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg\" (UID: \"f7dcbaab-2006-4334-b94d-33b242aac0ef\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg"
Nov 03 20:35:09.595622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:09.576436    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7dcbaab-2006-4334-b94d-33b242aac0ef-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg\" (UID: \"f7dcbaab-2006-4334-b94d-33b242aac0ef\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg"
Nov 03 20:35:09.809852 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:09.809774    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg"
Nov 03 20:35:10.143058 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:10.142981    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg"]
Nov 03 20:35:10.147939 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:35:10.147903    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7dcbaab_2006_4334_b94d_33b242aac0ef.slice/crio-f0006136a2e5ebe53005ef94b9b7af672b75b2689e8e559b4a76a355aac5061c WatchSource:0}: Error finding container f0006136a2e5ebe53005ef94b9b7af672b75b2689e8e559b4a76a355aac5061c: Status 404 returned error can't find the container with id f0006136a2e5ebe53005ef94b9b7af672b75b2689e8e559b4a76a355aac5061c
Nov 03 20:35:10.309558 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:10.309526    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg" event={"ID":"f7dcbaab-2006-4334-b94d-33b242aac0ef","Type":"ContainerStarted","Data":"db24d3b6e15f8d18f7ad06a20bfd89c0960d070aa2e490bcf66367cbd08d4647"}
Nov 03 20:35:10.309817 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:10.309578    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg" event={"ID":"f7dcbaab-2006-4334-b94d-33b242aac0ef","Type":"ContainerStarted","Data":"f0006136a2e5ebe53005ef94b9b7af672b75b2689e8e559b4a76a355aac5061c"}
Nov 03 20:35:10.309817 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:10.309666    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg"
Nov 03 20:35:10.331707 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:10.331661    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg" podStartSLOduration=1.331647476 podStartE2EDuration="1.331647476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:10.328527057 +0000 UTC m=+2040.461655535" watchObservedRunningTime="2026-11-03 20:35:10.331647476 +0000 UTC m=+2040.464775938"
Nov 03 20:35:11.284630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:11.277096    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th"]
Nov 03 20:35:11.331648 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:11.321977    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th"
Nov 03 20:35:11.331648 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:11.328953    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-6350b9\""
Nov 03 20:35:11.341740 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:11.341698    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th"]
Nov 03 20:35:11.401128 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:11.401085    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcd47c56-cc43-41ff-951b-82c5768ad799-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th\" (UID: \"bcd47c56-cc43-41ff-951b-82c5768ad799\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th"
Nov 03 20:35:11.401524 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:11.401500    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmcmv\" (UniqueName: \"kubernetes.io/projected/bcd47c56-cc43-41ff-951b-82c5768ad799-kube-api-access-hmcmv\") pod \"payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th\" (UID: \"bcd47c56-cc43-41ff-951b-82c5768ad799\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th"
Nov 03 20:35:11.514624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:11.512413    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmcmv\" (UniqueName: \"kubernetes.io/projected/bcd47c56-cc43-41ff-951b-82c5768ad799-kube-api-access-hmcmv\") pod \"payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th\" (UID: \"bcd47c56-cc43-41ff-951b-82c5768ad799\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th"
Nov 03 20:35:11.514624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:11.512629    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcd47c56-cc43-41ff-951b-82c5768ad799-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th\" (UID: \"bcd47c56-cc43-41ff-951b-82c5768ad799\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th"
Nov 03 20:35:11.548669 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:11.543166    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcd47c56-cc43-41ff-951b-82c5768ad799-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th\" (UID: \"bcd47c56-cc43-41ff-951b-82c5768ad799\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th"
Nov 03 20:35:11.592392 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:11.592317    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmcmv\" (UniqueName: \"kubernetes.io/projected/bcd47c56-cc43-41ff-951b-82c5768ad799-kube-api-access-hmcmv\") pod \"payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th\" (UID: \"bcd47c56-cc43-41ff-951b-82c5768ad799\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th"
Nov 03 20:35:11.671128 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:11.670894    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th"
Nov 03 20:35:12.077749 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:12.077714    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th"]
Nov 03 20:35:12.093953 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:35:12.083180    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd47c56_cc43_41ff_951b_82c5768ad799.slice/crio-4b1797ad25af337001a6daa4d42ea511e76629f18cba1563efba9973b95a5d0d WatchSource:0}: Error finding container 4b1797ad25af337001a6daa4d42ea511e76629f18cba1563efba9973b95a5d0d: Status 404 returned error can't find the container with id 4b1797ad25af337001a6daa4d42ea511e76629f18cba1563efba9973b95a5d0d
Nov 03 20:35:12.416846 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:12.415107    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th" event={"ID":"bcd47c56-cc43-41ff-951b-82c5768ad799","Type":"ContainerStarted","Data":"6f5a5b8322e6b543e74110dad9d2596379d45f9fa68b287df0b3227bfbd53175"}
Nov 03 20:35:12.416846 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:12.415172    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th" event={"ID":"bcd47c56-cc43-41ff-951b-82c5768ad799","Type":"ContainerStarted","Data":"4b1797ad25af337001a6daa4d42ea511e76629f18cba1563efba9973b95a5d0d"}
Nov 03 20:35:12.416846 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:12.416581    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th"
Nov 03 20:35:12.440791 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:12.440736    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th" podStartSLOduration=1.440722231 podStartE2EDuration="1.440722231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:12.434145517 +0000 UTC m=+2042.567273979" watchObservedRunningTime="2026-11-03 20:35:12.440722231 +0000 UTC m=+2042.573850708"
Nov 03 20:35:16.841188 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:16.841156    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg"]
Nov 03 20:35:16.844249 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:16.844220    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg"
Nov 03 20:35:19.631353 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:35:19.631313    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:35:21.841711 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:21.841672    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg" podUID="f7dcbaab-2006-4334-b94d-33b242aac0ef" containerName="payload-processing" containerID="cri-o://db24d3b6e15f8d18f7ad06a20bfd89c0960d070aa2e490bcf66367cbd08d4647" gracePeriod=25
Nov 03 20:35:22.141215 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:22.141187    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg"
Nov 03 20:35:22.162784 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:22.162760    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7dcbaab-2006-4334-b94d-33b242aac0ef-kube-api-access-xl6qc\" (UniqueName: \"kubernetes.io/projected/f7dcbaab-2006-4334-b94d-33b242aac0ef-kube-api-access-xl6qc\") pod \"f7dcbaab-2006-4334-b94d-33b242aac0ef\" (UID: \"f7dcbaab-2006-4334-b94d-33b242aac0ef\") "
Nov 03 20:35:22.162914 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:22.162884    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7dcbaab-2006-4334-b94d-33b242aac0ef-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7dcbaab-2006-4334-b94d-33b242aac0ef-provider-credentials\") pod \"f7dcbaab-2006-4334-b94d-33b242aac0ef\" (UID: \"f7dcbaab-2006-4334-b94d-33b242aac0ef\") "
Nov 03 20:35:22.162981 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:22.162951    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7dcbaab-2006-4334-b94d-33b242aac0ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7dcbaab-2006-4334-b94d-33b242aac0ef-plugins-config-volume\") pod \"f7dcbaab-2006-4334-b94d-33b242aac0ef\" (UID: \"f7dcbaab-2006-4334-b94d-33b242aac0ef\") "
Nov 03 20:35:22.163059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:22.163045    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7dcbaab-2006-4334-b94d-33b242aac0ef-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7dcbaab-2006-4334-b94d-33b242aac0ef-routing-overlay-volume\") pod \"f7dcbaab-2006-4334-b94d-33b242aac0ef\" (UID: \"f7dcbaab-2006-4334-b94d-33b242aac0ef\") "
Nov 03 20:35:22.163419 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:22.163349    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7dcbaab-2006-4334-b94d-33b242aac0ef-plugins-config-volume" pod "f7dcbaab-2006-4334-b94d-33b242aac0ef" (UID: "f7dcbaab-2006-4334-b94d-33b242aac0ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:35:22.163419 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:22.163383    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7dcbaab-2006-4334-b94d-33b242aac0ef-routing-overlay-volume" pod "f7dcbaab-2006-4334-b94d-33b242aac0ef" (UID: "f7dcbaab-2006-4334-b94d-33b242aac0ef"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:35:22.164129 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:22.164100    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7dcbaab-2006-4334-b94d-33b242aac0ef-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:22.164224 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:22.164133    4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7dcbaab-2006-4334-b94d-33b242aac0ef-routing-overlay-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:22.165355 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:22.165334    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7dcbaab-2006-4334-b94d-33b242aac0ef-kube-api-access-xl6qc" pod "f7dcbaab-2006-4334-b94d-33b242aac0ef" (UID: "f7dcbaab-2006-4334-b94d-33b242aac0ef"). InnerVolumeSpecName "kube-api-access-xl6qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:22.165437 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:22.165353    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7dcbaab-2006-4334-b94d-33b242aac0ef-provider-credentials" pod "f7dcbaab-2006-4334-b94d-33b242aac0ef" (UID: "f7dcbaab-2006-4334-b94d-33b242aac0ef"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:22.265830 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:22.265798    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl6qc\" (UniqueName: \"kubernetes.io/projected/f7dcbaab-2006-4334-b94d-33b242aac0ef-kube-api-access-xl6qc\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:22.265830 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:22.265827    4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7dcbaab-2006-4334-b94d-33b242aac0ef-provider-credentials\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:22.515135 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:22.515094    4421 generic.go:356] "Generic (PLEG): container finished" podID="f7dcbaab-2006-4334-b94d-33b242aac0ef" containerID="db24d3b6e15f8d18f7ad06a20bfd89c0960d070aa2e490bcf66367cbd08d4647" exitCode=0
Nov 03 20:35:22.515342 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:22.515158    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg"
Nov 03 20:35:22.515342 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:22.515161    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg" event={"ID":"f7dcbaab-2006-4334-b94d-33b242aac0ef","Type":"ContainerDied","Data":"db24d3b6e15f8d18f7ad06a20bfd89c0960d070aa2e490bcf66367cbd08d4647"}
Nov 03 20:35:22.515342 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:22.515239    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg" event={"ID":"f7dcbaab-2006-4334-b94d-33b242aac0ef","Type":"ContainerDied","Data":"f0006136a2e5ebe53005ef94b9b7af672b75b2689e8e559b4a76a355aac5061c"}
Nov 03 20:35:22.515342 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:22.515270    4421 scope.go:122] "RemoveContainer" containerID="db24d3b6e15f8d18f7ad06a20bfd89c0960d070aa2e490bcf66367cbd08d4647"
Nov 03 20:35:22.530088 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:22.530060    4421 scope.go:122] "RemoveContainer" containerID="db24d3b6e15f8d18f7ad06a20bfd89c0960d070aa2e490bcf66367cbd08d4647"
Nov 03 20:35:22.530443 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:35:22.530411    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db24d3b6e15f8d18f7ad06a20bfd89c0960d070aa2e490bcf66367cbd08d4647\": container with ID starting with db24d3b6e15f8d18f7ad06a20bfd89c0960d070aa2e490bcf66367cbd08d4647 not found: ID does not exist" containerID="db24d3b6e15f8d18f7ad06a20bfd89c0960d070aa2e490bcf66367cbd08d4647"
Nov 03 20:35:22.530522 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:22.530458    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db24d3b6e15f8d18f7ad06a20bfd89c0960d070aa2e490bcf66367cbd08d4647"} err="failed to get container status \"db24d3b6e15f8d18f7ad06a20bfd89c0960d070aa2e490bcf66367cbd08d4647\": rpc error: code = NotFound desc = could not find container \"db24d3b6e15f8d18f7ad06a20bfd89c0960d070aa2e490bcf66367cbd08d4647\": container with ID starting with db24d3b6e15f8d18f7ad06a20bfd89c0960d070aa2e490bcf66367cbd08d4647 not found: ID does not exist"
Nov 03 20:35:22.547358 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:22.547329    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg"]
Nov 03 20:35:22.554682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:22.554649    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-57c6qk8fg"]
Nov 03 20:35:24.020189 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:24.020156    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7dcbaab-2006-4334-b94d-33b242aac0ef" path="/var/lib/kubelet/pods/f7dcbaab-2006-4334-b94d-33b242aac0ef/volumes"
Nov 03 20:35:24.436119 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:24.436097    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th"
Nov 03 20:35:25.381454 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:25.381425    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8"]
Nov 03 20:35:29.116622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:29.098109    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t"]
Nov 03 20:35:29.116622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:29.099553    4421 cpu_manager.go:414] "containerMap: removing container" podUID="f7dcbaab-2006-4334-b94d-33b242aac0ef" containerName="payload-processing"
Nov 03 20:35:29.116622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:29.099576    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7dcbaab-2006-4334-b94d-33b242aac0ef" containerName="payload-processing"
Nov 03 20:35:29.116622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:29.099805    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7dcbaab-2006-4334-b94d-33b242aac0ef" containerName="payload-processing"
Nov 03 20:35:29.161746 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:29.161710    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t"
Nov 03 20:35:29.174729 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:29.161789    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t"]
Nov 03 20:35:29.273622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:29.268163    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-6350b9-serving-cert\""
Nov 03 20:35:29.283662 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:29.280069    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t\" (UID: \"c6e7fd23-0f9d-4ec7-9a57-1326c2a63287\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t"
Nov 03 20:35:29.283662 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:29.280268    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-metrics-certs\") pod \"maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t\" (UID: \"c6e7fd23-0f9d-4ec7-9a57-1326c2a63287\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t"
Nov 03 20:35:29.283662 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:29.280443    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t\" (UID: \"c6e7fd23-0f9d-4ec7-9a57-1326c2a63287\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t"
Nov 03 20:35:29.283662 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:29.280645    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zv9l\" (UniqueName: \"kubernetes.io/projected/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-kube-api-access-8zv9l\") pod \"maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t\" (UID: \"c6e7fd23-0f9d-4ec7-9a57-1326c2a63287\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t"
Nov 03 20:35:29.402675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:29.384455    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t\" (UID: \"c6e7fd23-0f9d-4ec7-9a57-1326c2a63287\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t"
Nov 03 20:35:29.402675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:29.384647    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zv9l\" (UniqueName: \"kubernetes.io/projected/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-kube-api-access-8zv9l\") pod \"maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t\" (UID: \"c6e7fd23-0f9d-4ec7-9a57-1326c2a63287\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t"
Nov 03 20:35:29.402675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:29.384767    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t\" (UID: \"c6e7fd23-0f9d-4ec7-9a57-1326c2a63287\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t"
Nov 03 20:35:29.402675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:29.384892    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-metrics-certs\") pod \"maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t\" (UID: \"c6e7fd23-0f9d-4ec7-9a57-1326c2a63287\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t"
Nov 03 20:35:29.402675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:29.386287    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t\" (UID: \"c6e7fd23-0f9d-4ec7-9a57-1326c2a63287\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t"
Nov 03 20:35:29.402675 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:35:29.386429    4421 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-serving-cert: secret "maas-api-e2e-extmod-b-6350b9-serving-cert" not found
Nov 03 20:35:29.402675 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:35:29.386484    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-maas-api-tls podName:c6e7fd23-0f9d-4ec7-9a57-1326c2a63287 nodeName:}" failed. No retries permitted until 2026-11-03 20:35:29.88646789 +0000 UTC m=+2060.019596343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-maas-api-tls") pod "maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t" (UID: "c6e7fd23-0f9d-4ec7-9a57-1326c2a63287") : secret "maas-api-e2e-extmod-b-6350b9-serving-cert" not found
Nov 03 20:35:29.461210 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:29.452585    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-metrics-certs\") pod \"maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t\" (UID: \"c6e7fd23-0f9d-4ec7-9a57-1326c2a63287\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t"
Nov 03 20:35:29.558049 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:29.558015    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zv9l\" (UniqueName: \"kubernetes.io/projected/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-kube-api-access-8zv9l\") pod \"maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t\" (UID: \"c6e7fd23-0f9d-4ec7-9a57-1326c2a63287\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t"
Nov 03 20:35:29.893633 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:29.893586    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t\" (UID: \"c6e7fd23-0f9d-4ec7-9a57-1326c2a63287\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t"
Nov 03 20:35:29.896348 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:29.896323    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t\" (UID: \"c6e7fd23-0f9d-4ec7-9a57-1326c2a63287\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t"
Nov 03 20:35:30.198635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:30.192919    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t"
Nov 03 20:35:30.382075 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:30.381969    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8" podUID="6e81b052-3cb9-47cf-9b09-67341b4deee1" containerName="payload-processing" containerID="cri-o://7db6b0d4b7be8d549a6d82b05afb80cbb6058e5f26879cbc322234f5122c302c" gracePeriod=25
Nov 03 20:35:30.692621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:30.689476    4421 generic.go:356] "Generic (PLEG): container finished" podID="6e81b052-3cb9-47cf-9b09-67341b4deee1" containerID="7db6b0d4b7be8d549a6d82b05afb80cbb6058e5f26879cbc322234f5122c302c" exitCode=0
Nov 03 20:35:30.692621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:30.689805    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8" event={"ID":"6e81b052-3cb9-47cf-9b09-67341b4deee1","Type":"ContainerDied","Data":"7db6b0d4b7be8d549a6d82b05afb80cbb6058e5f26879cbc322234f5122c302c"}
Nov 03 20:35:30.817743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:30.816365    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t"]
Nov 03 20:35:30.838402 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:35:30.837305    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e7fd23_0f9d_4ec7_9a57_1326c2a63287.slice/crio-0fb8f78f9bfb0a30f9e2bc0657c1e4706c9af086b4f155e24aa0c46f9e77272b WatchSource:0}: Error finding container 0fb8f78f9bfb0a30f9e2bc0657c1e4706c9af086b4f155e24aa0c46f9e77272b: Status 404 returned error can't find the container with id 0fb8f78f9bfb0a30f9e2bc0657c1e4706c9af086b4f155e24aa0c46f9e77272b
Nov 03 20:35:30.967882 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:30.967856    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8"
Nov 03 20:35:31.034720 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:31.034687    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e81b052-3cb9-47cf-9b09-67341b4deee1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e81b052-3cb9-47cf-9b09-67341b4deee1-routing-overlay-volume\") pod \"6e81b052-3cb9-47cf-9b09-67341b4deee1\" (UID: \"6e81b052-3cb9-47cf-9b09-67341b4deee1\") "
Nov 03 20:35:31.034937 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:31.034811    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e81b052-3cb9-47cf-9b09-67341b4deee1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e81b052-3cb9-47cf-9b09-67341b4deee1-provider-credentials\") pod \"6e81b052-3cb9-47cf-9b09-67341b4deee1\" (UID: \"6e81b052-3cb9-47cf-9b09-67341b4deee1\") "
Nov 03 20:35:31.034937 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:31.034929    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e81b052-3cb9-47cf-9b09-67341b4deee1-kube-api-access-8hhck\" (UniqueName: \"kubernetes.io/projected/6e81b052-3cb9-47cf-9b09-67341b4deee1-kube-api-access-8hhck\") pod \"6e81b052-3cb9-47cf-9b09-67341b4deee1\" (UID: \"6e81b052-3cb9-47cf-9b09-67341b4deee1\") "
Nov 03 20:35:31.035214 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:31.034973    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e81b052-3cb9-47cf-9b09-67341b4deee1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e81b052-3cb9-47cf-9b09-67341b4deee1-plugins-config-volume\") pod \"6e81b052-3cb9-47cf-9b09-67341b4deee1\" (UID: \"6e81b052-3cb9-47cf-9b09-67341b4deee1\") "
Nov 03 20:35:31.035214 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:31.035076    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e81b052-3cb9-47cf-9b09-67341b4deee1-routing-overlay-volume" pod "6e81b052-3cb9-47cf-9b09-67341b4deee1" (UID: "6e81b052-3cb9-47cf-9b09-67341b4deee1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:35:31.035922 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:31.035899    4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e81b052-3cb9-47cf-9b09-67341b4deee1-routing-overlay-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:31.036385 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:31.036355    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e81b052-3cb9-47cf-9b09-67341b4deee1-plugins-config-volume" pod "6e81b052-3cb9-47cf-9b09-67341b4deee1" (UID: "6e81b052-3cb9-47cf-9b09-67341b4deee1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:35:31.038133 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:31.038012    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e81b052-3cb9-47cf-9b09-67341b4deee1-kube-api-access-8hhck" pod "6e81b052-3cb9-47cf-9b09-67341b4deee1" (UID: "6e81b052-3cb9-47cf-9b09-67341b4deee1"). InnerVolumeSpecName "kube-api-access-8hhck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:31.038841 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:31.038747    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e81b052-3cb9-47cf-9b09-67341b4deee1-provider-credentials" pod "6e81b052-3cb9-47cf-9b09-67341b4deee1" (UID: "6e81b052-3cb9-47cf-9b09-67341b4deee1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:31.142674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:31.140405    4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e81b052-3cb9-47cf-9b09-67341b4deee1-provider-credentials\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:31.142674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:31.140452    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hhck\" (UniqueName: \"kubernetes.io/projected/6e81b052-3cb9-47cf-9b09-67341b4deee1-kube-api-access-8hhck\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:31.142674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:31.140467    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e81b052-3cb9-47cf-9b09-67341b4deee1-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:31.700892 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:31.700850    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t" event={"ID":"c6e7fd23-0f9d-4ec7-9a57-1326c2a63287","Type":"ContainerStarted","Data":"7708efa9313dcf3428eacb3c5b3399cae44f1c2f05bd0366e4e2068548a2a1c1"}
Nov 03 20:35:31.701397 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:31.700943    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t" event={"ID":"c6e7fd23-0f9d-4ec7-9a57-1326c2a63287","Type":"ContainerStarted","Data":"0fb8f78f9bfb0a30f9e2bc0657c1e4706c9af086b4f155e24aa0c46f9e77272b"}
Nov 03 20:35:31.701397 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:31.701368    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t"
Nov 03 20:35:31.703302 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:31.703274    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8"
Nov 03 20:35:31.703418 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:31.703334    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8" event={"ID":"6e81b052-3cb9-47cf-9b09-67341b4deee1","Type":"ContainerDied","Data":"737e922c3824c696920575cf7a00190e26c0fc9a266fd50b4423391f078750f7"}
Nov 03 20:35:31.703418 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:31.703394    4421 scope.go:122] "RemoveContainer" containerID="7db6b0d4b7be8d549a6d82b05afb80cbb6058e5f26879cbc322234f5122c302c"
Nov 03 20:35:31.726280 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:31.726233    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t" podStartSLOduration=3.277252355 podStartE2EDuration="3.72622051s" totalImagesPullingTime="448.968155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:31.721304913 +0000 UTC m=+2061.854433377" watchObservedRunningTime="2026-11-03 20:35:31.72622051 +0000 UTC m=+2061.859348998"
Nov 03 20:35:31.746652 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:31.746627    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8"]
Nov 03 20:35:31.751426 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:31.751399    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-6d65xmsd8"]
Nov 03 20:35:32.021875 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:32.021779    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e81b052-3cb9-47cf-9b09-67341b4deee1" path="/var/lib/kubelet/pods/6e81b052-3cb9-47cf-9b09-67341b4deee1/volumes"
Nov 03 20:35:37.719837 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:37.719809    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t"
Nov 03 20:35:46.201534 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.201498    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"]
Nov 03 20:35:46.202412 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.202394    4421 cpu_manager.go:414] "containerMap: removing container" podUID="6e81b052-3cb9-47cf-9b09-67341b4deee1" containerName="payload-processing"
Nov 03 20:35:46.202463 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.202414    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e81b052-3cb9-47cf-9b09-67341b4deee1" containerName="payload-processing"
Nov 03 20:35:46.202567 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.202558    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e81b052-3cb9-47cf-9b09-67341b4deee1" containerName="payload-processing"
Nov 03 20:35:46.207775 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.207756    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.212383 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.212356    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istiod-ca-cert\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.212632 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.212609    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-workload-certs\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.212785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.212765    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-podinfo\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.212908 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.212879    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-c7aecb-openshift-default-dockercfg-97wvf\""
Nov 03 20:35:46.213424 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.213402    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-envoy\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.213424 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.213589    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spdcb\" (UniqueName: \"kubernetes.io/projected/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-kube-api-access-spdcb\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.213424 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.213701    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-credential-socket\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.213424 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.213748    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-workload-socket\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.213424 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.213838    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-token\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.214215 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.213938    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-data\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.220283 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.220259    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"]
Nov 03 20:35:46.315029 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.314997    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spdcb\" (UniqueName: \"kubernetes.io/projected/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-kube-api-access-spdcb\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.315299 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.315273    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-credential-socket\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.315415 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.315325    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-workload-socket\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.315415 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.315364    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-token\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.315526 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.315430    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-data\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.315526 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.315471    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istiod-ca-cert\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.315676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.315568    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-workload-certs\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.315676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.315663    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-podinfo\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.315806 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.315716    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-credential-socket\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.315806 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.315796    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-workload-socket\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.315957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.315843    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-envoy\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.315957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.315938    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-data\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.316073 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.315996    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-workload-certs\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.316301 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.316281    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istiod-ca-cert\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.318383 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.318361    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-envoy\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.319102 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.319080    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-podinfo\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.341171 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.341149    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-token\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.341444 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.341422    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spdcb\" (UniqueName: \"kubernetes.io/projected/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-kube-api-access-spdcb\") pod \"e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") " pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.525723 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.525641    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:46.850518 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.850492    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"]
Nov 03 20:35:46.852500 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.852463    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:35:46.852657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.852545    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:35:46.852657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:46.852590    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:35:47.373376 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:47.373339    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm" event={"ID":"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae","Type":"ContainerStarted","Data":"ebf0b18190f0973556f120eebc7cc6284a7be162e983760cc68a4d1232324eb6"}
Nov 03 20:35:47.373801 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:47.373401    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm" event={"ID":"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae","Type":"ContainerStarted","Data":"5c2e35e7f96ffc2e84aeca33016304c74685c02546ebffd215972a08ddf72fe9"}
Nov 03 20:35:47.405571 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:47.405370    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm" podStartSLOduration=1.405351868 podStartE2EDuration="1.405351868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:47.398312625 +0000 UTC m=+2077.531441090" watchObservedRunningTime="2026-11-03 20:35:47.405351868 +0000 UTC m=+2077.538480330"
Nov 03 20:35:47.526662 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:47.526630    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:47.528117 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:47.528097    4421 patch_prober.go:29] interesting pod/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Nov 03 20:35:47.528203 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:47.528141    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm" podUID="b8cf0dee-6ac9-4932-9f4c-0d475d4597ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Nov 03 20:35:48.537948 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:48.537899    4421 patch_prober.go:29] interesting pod/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Nov 03 20:35:48.538520 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:48.538487    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm" podUID="b8cf0dee-6ac9-4932-9f4c-0d475d4597ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Nov 03 20:35:49.526450 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:49.526409    4421 patch_prober.go:29] interesting pod/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Nov 03 20:35:49.526674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:49.526466    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm" podUID="b8cf0dee-6ac9-4932-9f4c-0d475d4597ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Nov 03 20:35:49.632818 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:35:49.632783    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:35:50.526034 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:50.526003    4421 patch_prober.go:29] interesting pod/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Nov 03 20:35:50.526210 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:50.526054    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm" podUID="b8cf0dee-6ac9-4932-9f4c-0d475d4597ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Nov 03 20:35:51.526044 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:51.526008    4421 patch_prober.go:29] interesting pod/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Nov 03 20:35:51.526440 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:51.526063    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm" podUID="b8cf0dee-6ac9-4932-9f4c-0d475d4597ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Nov 03 20:35:52.526272 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:52.526233    4421 patch_prober.go:29] interesting pod/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Nov 03 20:35:52.526698 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:52.526291    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm" podUID="b8cf0dee-6ac9-4932-9f4c-0d475d4597ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Nov 03 20:35:53.532754 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:53.532726    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:53.533301 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:53.533039    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:53.533953 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:53.533933    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 20:35:58.330523 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:58.330490    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz"]
Nov 03 20:35:58.337047 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:58.337007    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz"
Nov 03 20:35:58.341857 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:58.341834    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6350b9\"/\"openshift-service-ca.crt\""
Nov 03 20:35:58.342057 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:58.341959    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6350b9\"/\"kube-root-ca.crt\""
Nov 03 20:35:58.342985 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:58.342965    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6350b9\"/\"payload-processing-external-model-plugins-e2e-extmod-b-6350b9\""
Nov 03 20:35:58.343638 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:58.343589    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz"]
Nov 03 20:35:58.343779 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:58.343759    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6350b9\"/\"agc-cred-5e9764f5fbfe\""
Nov 03 20:35:58.343876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:58.343762    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6350b9\"/\"routing-overlay\""
Nov 03 20:35:58.470622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:58.464941    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz\" (UID: \"4dee47b3-4df5-4b7b-bb29-c82ab58e2023\") " pod="ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz"
Nov 03 20:35:58.470622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:58.465133    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrhsv\" (UniqueName: \"kubernetes.io/projected/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-kube-api-access-mrhsv\") pod \"payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz\" (UID: \"4dee47b3-4df5-4b7b-bb29-c82ab58e2023\") " pod="ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz"
Nov 03 20:35:58.470622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:58.465356    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz\" (UID: \"4dee47b3-4df5-4b7b-bb29-c82ab58e2023\") " pod="ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz"
Nov 03 20:35:58.470622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:58.465469    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz\" (UID: \"4dee47b3-4df5-4b7b-bb29-c82ab58e2023\") " pod="ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz"
Nov 03 20:35:58.577139 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:58.577092    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz\" (UID: \"4dee47b3-4df5-4b7b-bb29-c82ab58e2023\") " pod="ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz"
Nov 03 20:35:58.577441 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:58.577420    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz\" (UID: \"4dee47b3-4df5-4b7b-bb29-c82ab58e2023\") " pod="ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz"
Nov 03 20:35:58.577709 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:58.577691    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz\" (UID: \"4dee47b3-4df5-4b7b-bb29-c82ab58e2023\") " pod="ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz"
Nov 03 20:35:58.587908 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:58.587819    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrhsv\" (UniqueName: \"kubernetes.io/projected/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-kube-api-access-mrhsv\") pod \"payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz\" (UID: \"4dee47b3-4df5-4b7b-bb29-c82ab58e2023\") " pod="ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz"
Nov 03 20:35:58.605636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:58.598408    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz\" (UID: \"4dee47b3-4df5-4b7b-bb29-c82ab58e2023\") " pod="ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz"
Nov 03 20:35:58.606577 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:58.606545    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz\" (UID: \"4dee47b3-4df5-4b7b-bb29-c82ab58e2023\") " pod="ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz"
Nov 03 20:35:58.628682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:58.628207    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz\" (UID: \"4dee47b3-4df5-4b7b-bb29-c82ab58e2023\") " pod="ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz"
Nov 03 20:35:58.682491 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:58.682420    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrhsv\" (UniqueName: \"kubernetes.io/projected/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-kube-api-access-mrhsv\") pod \"payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz\" (UID: \"4dee47b3-4df5-4b7b-bb29-c82ab58e2023\") " pod="ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz"
Nov 03 20:35:58.690631 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:58.689088    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz"
Nov 03 20:35:59.148876 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:35:59.148844    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dee47b3_4df5_4b7b_bb29_c82ab58e2023.slice/crio-d0389b1e15639096acff4eae3f78f4386e6f83a83561a443a8e42cab7f0662d4 WatchSource:0}: Error finding container d0389b1e15639096acff4eae3f78f4386e6f83a83561a443a8e42cab7f0662d4: Status 404 returned error can't find the container with id d0389b1e15639096acff4eae3f78f4386e6f83a83561a443a8e42cab7f0662d4
Nov 03 20:35:59.149509 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:59.149481    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz"]
Nov 03 20:35:59.597349 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:59.597312    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz" event={"ID":"4dee47b3-4df5-4b7b-bb29-c82ab58e2023","Type":"ContainerStarted","Data":"5b9e137fca15aef527736a8a1645787dedcae6a06193bfe12244d9b1822ec6da"}
Nov 03 20:35:59.597818 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:59.597374    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz" event={"ID":"4dee47b3-4df5-4b7b-bb29-c82ab58e2023","Type":"ContainerStarted","Data":"d0389b1e15639096acff4eae3f78f4386e6f83a83561a443a8e42cab7f0662d4"}
Nov 03 20:35:59.597818 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:59.597459    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz"
Nov 03 20:35:59.622338 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:35:59.622282    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz" podStartSLOduration=1.622268976 podStartE2EDuration="1.622268976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:59.617660128 +0000 UTC m=+2089.750788590" watchObservedRunningTime="2026-11-03 20:35:59.622268976 +0000 UTC m=+2089.755397471"
Nov 03 20:36:10.631623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:10.628801    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz"
Nov 03 20:36:14.828324 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:14.828289    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n"]
Nov 03 20:36:14.834578 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:14.834557    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n"
Nov 03 20:36:14.840024 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:14.840000    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"agc-cred-e4b314c05477\""
Nov 03 20:36:14.840245 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:14.840011    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"kube-root-ca.crt\""
Nov 03 20:36:14.840379 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:14.840176    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"payload-processing-external-model-plugins-e2e-extmod-a-db5198\""
Nov 03 20:36:14.841090 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:14.841071    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"openshift-service-ca.crt\""
Nov 03 20:36:14.841293 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:14.841277    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"routing-overlay\""
Nov 03 20:36:14.841386 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:14.841350    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-db5198\"/\"payload-processing-external-model-e2e--e92303d8-dockercfg-sgvcw\""
Nov 03 20:36:14.846664 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:14.846640    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n"]
Nov 03 20:36:14.909620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:14.902848    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b49ddb4-a348-449f-8899-5c7afb25115f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n\" (UID: \"4b49ddb4-a348-449f-8899-5c7afb25115f\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n"
Nov 03 20:36:14.909620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:14.902941    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8gzd\" (UniqueName: \"kubernetes.io/projected/4b49ddb4-a348-449f-8899-5c7afb25115f-kube-api-access-r8gzd\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n\" (UID: \"4b49ddb4-a348-449f-8899-5c7afb25115f\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n"
Nov 03 20:36:14.909620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:14.902988    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4b49ddb4-a348-449f-8899-5c7afb25115f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n\" (UID: \"4b49ddb4-a348-449f-8899-5c7afb25115f\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n"
Nov 03 20:36:14.909620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:14.903028    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4b49ddb4-a348-449f-8899-5c7afb25115f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n\" (UID: \"4b49ddb4-a348-449f-8899-5c7afb25115f\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n"
Nov 03 20:36:15.129675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:15.128886    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b49ddb4-a348-449f-8899-5c7afb25115f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n\" (UID: \"4b49ddb4-a348-449f-8899-5c7afb25115f\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n"
Nov 03 20:36:15.129675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:15.128995    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8gzd\" (UniqueName: \"kubernetes.io/projected/4b49ddb4-a348-449f-8899-5c7afb25115f-kube-api-access-r8gzd\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n\" (UID: \"4b49ddb4-a348-449f-8899-5c7afb25115f\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n"
Nov 03 20:36:15.129675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:15.129052    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4b49ddb4-a348-449f-8899-5c7afb25115f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n\" (UID: \"4b49ddb4-a348-449f-8899-5c7afb25115f\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n"
Nov 03 20:36:15.129675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:15.129096    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4b49ddb4-a348-449f-8899-5c7afb25115f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n\" (UID: \"4b49ddb4-a348-449f-8899-5c7afb25115f\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n"
Nov 03 20:36:15.141948 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:15.130209    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4b49ddb4-a348-449f-8899-5c7afb25115f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n\" (UID: \"4b49ddb4-a348-449f-8899-5c7afb25115f\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n"
Nov 03 20:36:15.141948 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:15.131314    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b49ddb4-a348-449f-8899-5c7afb25115f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n\" (UID: \"4b49ddb4-a348-449f-8899-5c7afb25115f\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n"
Nov 03 20:36:15.228233 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:15.202301    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4b49ddb4-a348-449f-8899-5c7afb25115f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n\" (UID: \"4b49ddb4-a348-449f-8899-5c7afb25115f\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n"
Nov 03 20:36:15.289080 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:15.288336    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8gzd\" (UniqueName: \"kubernetes.io/projected/4b49ddb4-a348-449f-8899-5c7afb25115f-kube-api-access-r8gzd\") pod \"payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n\" (UID: \"4b49ddb4-a348-449f-8899-5c7afb25115f\") " pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n"
Nov 03 20:36:15.455267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:15.455234    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n"
Nov 03 20:36:15.851168 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:15.851091    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n"]
Nov 03 20:36:16.867742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:16.867710    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n" event={"ID":"4b49ddb4-a348-449f-8899-5c7afb25115f","Type":"ContainerStarted","Data":"9ae79e8de8bd63521d257d8ef0968ad1a5cf9e6b42a7f860a8158e961d3ff65f"}
Nov 03 20:36:16.868161 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:16.867764    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n" event={"ID":"4b49ddb4-a348-449f-8899-5c7afb25115f","Type":"ContainerStarted","Data":"833c782309e24c8e882574d3d4b85512ba757763694914b8e6d6ab081234ec56"}
Nov 03 20:36:16.868161 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:16.867791    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n"
Nov 03 20:36:16.889879 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:16.889829    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n" podStartSLOduration=2.889814049 podStartE2EDuration="2.889814049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:16.887978728 +0000 UTC m=+2107.021107188" watchObservedRunningTime="2026-11-03 20:36:16.889814049 +0000 UTC m=+2107.022942537"
Nov 03 20:36:19.630298 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:36:19.630208    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:36:21.725365 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:21.725332    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n"]
Nov 03 20:36:21.728285 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:21.728264    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n"
Nov 03 20:36:26.725769 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:26.725719    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n" podUID="4b49ddb4-a348-449f-8899-5c7afb25115f" containerName="payload-processing" containerID="cri-o://9ae79e8de8bd63521d257d8ef0968ad1a5cf9e6b42a7f860a8158e961d3ff65f" gracePeriod=25
Nov 03 20:36:27.006550 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:27.006518    4421 generic.go:356] "Generic (PLEG): container finished" podID="4b49ddb4-a348-449f-8899-5c7afb25115f" containerID="9ae79e8de8bd63521d257d8ef0968ad1a5cf9e6b42a7f860a8158e961d3ff65f" exitCode=0
Nov 03 20:36:27.006707 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:27.006589    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n" event={"ID":"4b49ddb4-a348-449f-8899-5c7afb25115f","Type":"ContainerDied","Data":"9ae79e8de8bd63521d257d8ef0968ad1a5cf9e6b42a7f860a8158e961d3ff65f"}
Nov 03 20:36:27.006707 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:27.006670    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n" event={"ID":"4b49ddb4-a348-449f-8899-5c7afb25115f","Type":"ContainerDied","Data":"833c782309e24c8e882574d3d4b85512ba757763694914b8e6d6ab081234ec56"}
Nov 03 20:36:27.006707 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:27.006687    4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="833c782309e24c8e882574d3d4b85512ba757763694914b8e6d6ab081234ec56"
Nov 03 20:36:27.075586 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:27.075188    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n"
Nov 03 20:36:27.114675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:27.112852    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b49ddb4-a348-449f-8899-5c7afb25115f-kube-api-access-r8gzd\" (UniqueName: \"kubernetes.io/projected/4b49ddb4-a348-449f-8899-5c7afb25115f-kube-api-access-r8gzd\") pod \"4b49ddb4-a348-449f-8899-5c7afb25115f\" (UID: \"4b49ddb4-a348-449f-8899-5c7afb25115f\") "
Nov 03 20:36:27.114675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:27.113226    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b49ddb4-a348-449f-8899-5c7afb25115f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b49ddb4-a348-449f-8899-5c7afb25115f-plugins-config-volume\") pod \"4b49ddb4-a348-449f-8899-5c7afb25115f\" (UID: \"4b49ddb4-a348-449f-8899-5c7afb25115f\") "
Nov 03 20:36:27.114675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:27.113322    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b49ddb4-a348-449f-8899-5c7afb25115f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4b49ddb4-a348-449f-8899-5c7afb25115f-routing-overlay-volume\") pod \"4b49ddb4-a348-449f-8899-5c7afb25115f\" (UID: \"4b49ddb4-a348-449f-8899-5c7afb25115f\") "
Nov 03 20:36:27.114675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:27.113394    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b49ddb4-a348-449f-8899-5c7afb25115f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4b49ddb4-a348-449f-8899-5c7afb25115f-provider-credentials\") pod \"4b49ddb4-a348-449f-8899-5c7afb25115f\" (UID: \"4b49ddb4-a348-449f-8899-5c7afb25115f\") "
Nov 03 20:36:27.118627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:27.117977    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b49ddb4-a348-449f-8899-5c7afb25115f-routing-overlay-volume" pod "4b49ddb4-a348-449f-8899-5c7afb25115f" (UID: "4b49ddb4-a348-449f-8899-5c7afb25115f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:36:27.128571 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:27.126847    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b49ddb4-a348-449f-8899-5c7afb25115f-plugins-config-volume" pod "4b49ddb4-a348-449f-8899-5c7afb25115f" (UID: "4b49ddb4-a348-449f-8899-5c7afb25115f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:36:27.139926 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:27.139889    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b49ddb4-a348-449f-8899-5c7afb25115f-kube-api-access-r8gzd" pod "4b49ddb4-a348-449f-8899-5c7afb25115f" (UID: "4b49ddb4-a348-449f-8899-5c7afb25115f"). InnerVolumeSpecName "kube-api-access-r8gzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:27.140088 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:27.139951    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b49ddb4-a348-449f-8899-5c7afb25115f-provider-credentials" pod "4b49ddb4-a348-449f-8899-5c7afb25115f" (UID: "4b49ddb4-a348-449f-8899-5c7afb25115f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:27.215647 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:27.215582    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b49ddb4-a348-449f-8899-5c7afb25115f-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:27.215647 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:27.215641    4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4b49ddb4-a348-449f-8899-5c7afb25115f-routing-overlay-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:27.215647 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:27.215651    4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4b49ddb4-a348-449f-8899-5c7afb25115f-provider-credentials\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:27.215894 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:27.215660    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8gzd\" (UniqueName: \"kubernetes.io/projected/4b49ddb4-a348-449f-8899-5c7afb25115f-kube-api-access-r8gzd\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:28.015213 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:28.015181    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n"
Nov 03 20:36:28.043784 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:28.043746    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n"]
Nov 03 20:36:28.048936 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:28.048913    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-db5198/payload-processing-external-model-e2e-extmod-a-db5198-64cf9z45n"]
Nov 03 20:36:30.018757 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:30.018727    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b49ddb4-a348-449f-8899-5c7afb25115f" path="/var/lib/kubelet/pods/4b49ddb4-a348-449f-8899-5c7afb25115f/volumes"
Nov 03 20:36:39.337785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:39.337736    4421 patch_prober.go:29] interesting pod/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:36:39.338195 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:39.337807    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm" podUID="56233474-fdc7-4bcb-aa2a-db87ac074168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:36:42.941172 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:36:42.940570    4421 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56233474_fdc7_4bcb_aa2a_db87ac074168.slice/crio-conmon-f509b2406b74172416939d17669932f60e42e4b09fab9ec1df4f36089134c444.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:36:42.941172 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:36:42.940572    4421 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56233474_fdc7_4bcb_aa2a_db87ac074168.slice/crio-f509b2406b74172416939d17669932f60e42e4b09fab9ec1df4f36089134c444.scope\": RecentStats: unable to find data in memory cache], [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@20-33661-0.service\": RecentStats: unable to find data in memory cache]"
Nov 03 20:36:43.301855 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:43.301821    4421 generic.go:356] "Generic (PLEG): container finished" podID="56233474-fdc7-4bcb-aa2a-db87ac074168" containerID="f509b2406b74172416939d17669932f60e42e4b09fab9ec1df4f36089134c444" exitCode=0
Nov 03 20:36:43.302044 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:43.301870    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm" event={"ID":"56233474-fdc7-4bcb-aa2a-db87ac074168","Type":"ContainerDied","Data":"f509b2406b74172416939d17669932f60e42e4b09fab9ec1df4f36089134c444"}
Nov 03 20:36:43.302357 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:43.302338    4421 scope.go:122] "RemoveContainer" containerID="f509b2406b74172416939d17669932f60e42e4b09fab9ec1df4f36089134c444"
Nov 03 20:36:43.304425 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:43.304389    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:36:43.304546 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:43.304484    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:36:43.304546 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:43.304535    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:36:44.312182 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:44.312149    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm" event={"ID":"56233474-fdc7-4bcb-aa2a-db87ac074168","Type":"ContainerStarted","Data":"3c03823108946e2e3da7e4c7ca1035410a393f205ea01f13e56fd81c283fdb92"}
Nov 03 20:36:44.461042 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:44.461005    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:36:44.461452 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:44.461351    4421 patch_prober.go:29] interesting pod/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Nov 03 20:36:44.461662 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:44.461638    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm" podUID="56233474-fdc7-4bcb-aa2a-db87ac074168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Nov 03 20:36:45.466351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:45.466010    4421 patch_prober.go:29] interesting pod/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Nov 03 20:36:45.466351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:45.466060    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm" podUID="56233474-fdc7-4bcb-aa2a-db87ac074168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Nov 03 20:36:46.452626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:46.449020    4421 patch_prober.go:29] interesting pod/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Nov 03 20:36:46.452626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:46.449096    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm" podUID="56233474-fdc7-4bcb-aa2a-db87ac074168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Nov 03 20:36:47.464044 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:47.464007    4421 patch_prober.go:29] interesting pod/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Nov 03 20:36:47.464684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:47.464646    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm" podUID="56233474-fdc7-4bcb-aa2a-db87ac074168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Nov 03 20:36:48.467454 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:48.467417    4421 patch_prober.go:29] interesting pod/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Nov 03 20:36:48.468064 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:48.468030    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm" podUID="56233474-fdc7-4bcb-aa2a-db87ac074168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Nov 03 20:36:49.462623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:49.452981    4421 patch_prober.go:29] interesting pod/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Nov 03 20:36:49.462623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:49.453043    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm" podUID="56233474-fdc7-4bcb-aa2a-db87ac074168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Nov 03 20:36:49.631105 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:36:49.631073    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:36:50.454473 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:50.454447    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:36:50.454939 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:50.454927    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:36:50.455664 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:36:50.455642    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 20:37:14.263920 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:14.263255    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf"]
Nov 03 20:37:14.264838 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:14.264644    4421 cpu_manager.go:414] "containerMap: removing container" podUID="4b49ddb4-a348-449f-8899-5c7afb25115f" containerName="payload-processing"
Nov 03 20:37:14.264838 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:14.264668    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b49ddb4-a348-449f-8899-5c7afb25115f" containerName="payload-processing"
Nov 03 20:37:14.265131 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:14.264912    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b49ddb4-a348-449f-8899-5c7afb25115f" containerName="payload-processing"
Nov 03 20:37:14.314789 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:14.299998    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf"
Nov 03 20:37:14.316339 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:14.304079    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf"]
Nov 03 20:37:14.374351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:14.331550    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-05473b\""
Nov 03 20:37:14.374351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:14.331682    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"agc-cred-b7e8ea2ec1e9\""
Nov 03 20:37:14.374351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:14.331753    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"openshift-service-ca.crt\""
Nov 03 20:37:14.374351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:14.331847    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"agc-cred-0d43e4ffb5b7\""
Nov 03 20:37:14.394882 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:14.394851    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"kube-root-ca.crt\""
Nov 03 20:37:14.395424 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:14.395405    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"routing-overlay\""
Nov 03 20:37:14.395715 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:14.395696    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"payload-processing-external-model-e2e--f590b3a5-dockercfg-mjjk7\""
Nov 03 20:37:14.410541 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:14.410487    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf\" (UID: \"8d1d77d9-42e6-4eac-a09b-4cad11a204ba\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf"
Nov 03 20:37:14.410770 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:14.410584    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf\" (UID: \"8d1d77d9-42e6-4eac-a09b-4cad11a204ba\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf"
Nov 03 20:37:14.410770 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:14.410729    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7gfv\" (UniqueName: \"kubernetes.io/projected/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-kube-api-access-r7gfv\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf\" (UID: \"8d1d77d9-42e6-4eac-a09b-4cad11a204ba\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf"
Nov 03 20:37:14.410920 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:14.410903    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf\" (UID: \"8d1d77d9-42e6-4eac-a09b-4cad11a204ba\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf"
Nov 03 20:37:14.516676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:14.516045    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf\" (UID: \"8d1d77d9-42e6-4eac-a09b-4cad11a204ba\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf"
Nov 03 20:37:14.516676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:14.516368    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf\" (UID: \"8d1d77d9-42e6-4eac-a09b-4cad11a204ba\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf"
Nov 03 20:37:14.516676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:14.516405    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf\" (UID: \"8d1d77d9-42e6-4eac-a09b-4cad11a204ba\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf"
Nov 03 20:37:14.516676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:14.516471    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7gfv\" (UniqueName: \"kubernetes.io/projected/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-kube-api-access-r7gfv\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf\" (UID: \"8d1d77d9-42e6-4eac-a09b-4cad11a204ba\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf"
Nov 03 20:37:14.583403 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:14.583368    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf\" (UID: \"8d1d77d9-42e6-4eac-a09b-4cad11a204ba\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf"
Nov 03 20:37:14.584179 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:14.584155    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf\" (UID: \"8d1d77d9-42e6-4eac-a09b-4cad11a204ba\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf"
Nov 03 20:37:14.624067 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:14.623139    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7gfv\" (UniqueName: \"kubernetes.io/projected/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-kube-api-access-r7gfv\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf\" (UID: \"8d1d77d9-42e6-4eac-a09b-4cad11a204ba\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf"
Nov 03 20:37:14.670805 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:14.670767    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf\" (UID: \"8d1d77d9-42e6-4eac-a09b-4cad11a204ba\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf"
Nov 03 20:37:14.714712 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:14.714682    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf"
Nov 03 20:37:15.005853 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:37:15.005819    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1d77d9_42e6_4eac_a09b_4cad11a204ba.slice/crio-4a262b77de91ed8e9d062c1324015005faaeb647b1ecfde2c9952166a21eaea2 WatchSource:0}: Error finding container 4a262b77de91ed8e9d062c1324015005faaeb647b1ecfde2c9952166a21eaea2: Status 404 returned error can't find the container with id 4a262b77de91ed8e9d062c1324015005faaeb647b1ecfde2c9952166a21eaea2
Nov 03 20:37:15.007836 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:15.007807    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf"]
Nov 03 20:37:15.945963 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:15.945927    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf" event={"ID":"8d1d77d9-42e6-4eac-a09b-4cad11a204ba","Type":"ContainerStarted","Data":"7cc3f01ea4dabec3656e227577cfc3067facdfed14f0502a095d76ef2ab789c0"}
Nov 03 20:37:15.946438 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:15.945983    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf" event={"ID":"8d1d77d9-42e6-4eac-a09b-4cad11a204ba","Type":"ContainerStarted","Data":"4a262b77de91ed8e9d062c1324015005faaeb647b1ecfde2c9952166a21eaea2"}
Nov 03 20:37:15.946438 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:15.946045    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf"
Nov 03 20:37:15.977408 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:15.977359    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf" podStartSLOduration=1.977343691 podStartE2EDuration="1.977343691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:15.974429406 +0000 UTC m=+2166.107557868" watchObservedRunningTime="2026-11-03 20:37:15.977343691 +0000 UTC m=+2166.110472153"
Nov 03 20:37:19.631187 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:37:19.631152    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:37:26.572714 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:26.560000    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf"]
Nov 03 20:37:26.791630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:26.776228    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf"
Nov 03 20:37:30.638912 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:30.638873    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"]
Nov 03 20:37:30.639351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:30.639066    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5" podUID="989c3294-e402-4c4d-8369-ca96d89fa5cd" containerName="istio-proxy" containerID="cri-o://21eac988b1a95ccb158ef275484c9b99b715b60fbb79d72d5c59e8004e70ce71" gracePeriod=30
Nov 03 20:37:31.580368 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:31.580328    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf" podUID="8d1d77d9-42e6-4eac-a09b-4cad11a204ba" containerName="payload-processing" containerID="cri-o://7cc3f01ea4dabec3656e227577cfc3067facdfed14f0502a095d76ef2ab789c0" gracePeriod=25
Nov 03 20:37:31.877841 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:31.877817    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf"
Nov 03 20:37:32.017819 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.017785    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-kube-api-access-r7gfv\" (UniqueName: \"kubernetes.io/projected/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-kube-api-access-r7gfv\") pod \"8d1d77d9-42e6-4eac-a09b-4cad11a204ba\" (UID: \"8d1d77d9-42e6-4eac-a09b-4cad11a204ba\") "
Nov 03 20:37:32.019654 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.018061    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-plugins-config-volume\") pod \"8d1d77d9-42e6-4eac-a09b-4cad11a204ba\" (UID: \"8d1d77d9-42e6-4eac-a09b-4cad11a204ba\") "
Nov 03 20:37:32.019654 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.018201    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-provider-credentials\") pod \"8d1d77d9-42e6-4eac-a09b-4cad11a204ba\" (UID: \"8d1d77d9-42e6-4eac-a09b-4cad11a204ba\") "
Nov 03 20:37:32.019654 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.018463    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-routing-overlay-volume\") pod \"8d1d77d9-42e6-4eac-a09b-4cad11a204ba\" (UID: \"8d1d77d9-42e6-4eac-a09b-4cad11a204ba\") "
Nov 03 20:37:32.020663 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.020632    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-plugins-config-volume" pod "8d1d77d9-42e6-4eac-a09b-4cad11a204ba" (UID: "8d1d77d9-42e6-4eac-a09b-4cad11a204ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:32.032628 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.024144    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-routing-overlay-volume" pod "8d1d77d9-42e6-4eac-a09b-4cad11a204ba" (UID: "8d1d77d9-42e6-4eac-a09b-4cad11a204ba"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:32.096863 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.095403    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-kube-api-access-r7gfv" pod "8d1d77d9-42e6-4eac-a09b-4cad11a204ba" (UID: "8d1d77d9-42e6-4eac-a09b-4cad11a204ba"). InnerVolumeSpecName "kube-api-access-r7gfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:32.108365 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.108056    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-provider-credentials" pod "8d1d77d9-42e6-4eac-a09b-4cad11a204ba" (UID: "8d1d77d9-42e6-4eac-a09b-4cad11a204ba"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:32.123212 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.123184    4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-routing-overlay-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:32.123212 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.123213    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7gfv\" (UniqueName: \"kubernetes.io/projected/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-kube-api-access-r7gfv\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:32.123359 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.123222    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:32.123359 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.123232    4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d1d77d9-42e6-4eac-a09b-4cad11a204ba-provider-credentials\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:32.183667 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.183618    4421 generic.go:356] "Generic (PLEG): container finished" podID="8d1d77d9-42e6-4eac-a09b-4cad11a204ba" containerID="7cc3f01ea4dabec3656e227577cfc3067facdfed14f0502a095d76ef2ab789c0" exitCode=0
Nov 03 20:37:32.183873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.183688    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf"
Nov 03 20:37:32.183873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.183699    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf" event={"ID":"8d1d77d9-42e6-4eac-a09b-4cad11a204ba","Type":"ContainerDied","Data":"7cc3f01ea4dabec3656e227577cfc3067facdfed14f0502a095d76ef2ab789c0"}
Nov 03 20:37:32.183873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.183772    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf" event={"ID":"8d1d77d9-42e6-4eac-a09b-4cad11a204ba","Type":"ContainerDied","Data":"4a262b77de91ed8e9d062c1324015005faaeb647b1ecfde2c9952166a21eaea2"}
Nov 03 20:37:32.183873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.183805    4421 scope.go:122] "RemoveContainer" containerID="7cc3f01ea4dabec3656e227577cfc3067facdfed14f0502a095d76ef2ab789c0"
Nov 03 20:37:32.194772 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.194752    4421 scope.go:122] "RemoveContainer" containerID="7cc3f01ea4dabec3656e227577cfc3067facdfed14f0502a095d76ef2ab789c0"
Nov 03 20:37:32.195083 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:37:32.195064    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cc3f01ea4dabec3656e227577cfc3067facdfed14f0502a095d76ef2ab789c0\": container with ID starting with 7cc3f01ea4dabec3656e227577cfc3067facdfed14f0502a095d76ef2ab789c0 not found: ID does not exist" containerID="7cc3f01ea4dabec3656e227577cfc3067facdfed14f0502a095d76ef2ab789c0"
Nov 03 20:37:32.195139 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.195091    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cc3f01ea4dabec3656e227577cfc3067facdfed14f0502a095d76ef2ab789c0"} err="failed to get container status \"7cc3f01ea4dabec3656e227577cfc3067facdfed14f0502a095d76ef2ab789c0\": rpc error: code = NotFound desc = could not find container \"7cc3f01ea4dabec3656e227577cfc3067facdfed14f0502a095d76ef2ab789c0\": container with ID starting with 7cc3f01ea4dabec3656e227577cfc3067facdfed14f0502a095d76ef2ab789c0 not found: ID does not exist"
Nov 03 20:37:32.207666 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.207630    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf"]
Nov 03 20:37:32.212360 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.212319    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-d48b669lf"]
Nov 03 20:37:32.797678 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.797643    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"]
Nov 03 20:37:32.798652 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.798578    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw" podUID="e840e124-446a-4ef7-8ed3-abc704048d25" containerName="istio-proxy" containerID="cri-o://a545fe0897cffeef80b749e593bc1323b971d280fda0d0709373c4ceaa0bd31c" gracePeriod=30
Nov 03 20:37:32.811797 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.811763    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"]
Nov 03 20:37:32.812322 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.812286    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b" podUID="3844071e-3d45-455b-91be-3de079eacafa" containerName="istio-proxy" containerID="cri-o://bf94f44daf6fa955fb98218662473ead8be0652518b77894f77d37dff817a3b1" gracePeriod=30
Nov 03 20:37:32.822020 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.821996    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"]
Nov 03 20:37:32.822423 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.822394    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4" podUID="3d2b0564-e201-4bab-9dec-422ecc7955b1" containerName="istio-proxy" containerID="cri-o://ac70a7a5a44774198c2eaf131db35305d9d0f77dfefb7ebf50f57a271607ab65" gracePeriod=30
Nov 03 20:37:32.831266 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.831238    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"]
Nov 03 20:37:32.831581 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:32.831555    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt" podUID="5a14b4da-8ce7-457e-89b8-6e4788d924f0" containerName="istio-proxy" containerID="cri-o://ec9d322207b181bfaba3248655446b57cc3138cd2dd31cd41e0cbe3b529d6eff" gracePeriod=30
Nov 03 20:37:34.024481 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:34.024449    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d1d77d9-42e6-4eac-a09b-4cad11a204ba" path="/var/lib/kubelet/pods/8d1d77d9-42e6-4eac-a09b-4cad11a204ba/volumes"
Nov 03 20:37:36.033253 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.033221    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:37:36.088525 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.088487    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-workload-certs\") pod \"989c3294-e402-4c4d-8369-ca96d89fa5cd\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") "
Nov 03 20:37:36.088773 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.088676    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-envoy\") pod \"989c3294-e402-4c4d-8369-ca96d89fa5cd\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") "
Nov 03 20:37:36.088773 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.088741    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-token\" (UniqueName: \"kubernetes.io/projected/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-token\") pod \"989c3294-e402-4c4d-8369-ca96d89fa5cd\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") "
Nov 03 20:37:36.088911 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.088829    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-workload-certs" pod "989c3294-e402-4c4d-8369-ca96d89fa5cd" (UID: "989c3294-e402-4c4d-8369-ca96d89fa5cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:36.088911 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.088850    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-podinfo\") pod \"989c3294-e402-4c4d-8369-ca96d89fa5cd\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") "
Nov 03 20:37:36.089073 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.088965    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-workload-socket\") pod \"989c3294-e402-4c4d-8369-ca96d89fa5cd\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") "
Nov 03 20:37:36.089073 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.089040    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-credential-socket\") pod \"989c3294-e402-4c4d-8369-ca96d89fa5cd\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") "
Nov 03 20:37:36.089180 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.089070    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-data\") pod \"989c3294-e402-4c4d-8369-ca96d89fa5cd\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") "
Nov 03 20:37:36.089180 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.089113    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/989c3294-e402-4c4d-8369-ca96d89fa5cd-kube-api-access-2gl7g\" (UniqueName: \"kubernetes.io/projected/989c3294-e402-4c4d-8369-ca96d89fa5cd-kube-api-access-2gl7g\") pod \"989c3294-e402-4c4d-8369-ca96d89fa5cd\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") "
Nov 03 20:37:36.089295 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.089192    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/989c3294-e402-4c4d-8369-ca96d89fa5cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/989c3294-e402-4c4d-8369-ca96d89fa5cd-istiod-ca-cert\") pod \"989c3294-e402-4c4d-8369-ca96d89fa5cd\" (UID: \"989c3294-e402-4c4d-8369-ca96d89fa5cd\") "
Nov 03 20:37:36.090516 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.089385    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-data" pod "989c3294-e402-4c4d-8369-ca96d89fa5cd" (UID: "989c3294-e402-4c4d-8369-ca96d89fa5cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:36.090516 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.089773    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-workload-socket" pod "989c3294-e402-4c4d-8369-ca96d89fa5cd" (UID: "989c3294-e402-4c4d-8369-ca96d89fa5cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:36.090516 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.089882    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-credential-socket" pod "989c3294-e402-4c4d-8369-ca96d89fa5cd" (UID: "989c3294-e402-4c4d-8369-ca96d89fa5cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:36.090516 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.089800    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/989c3294-e402-4c4d-8369-ca96d89fa5cd-istiod-ca-cert" pod "989c3294-e402-4c4d-8369-ca96d89fa5cd" (UID: "989c3294-e402-4c4d-8369-ca96d89fa5cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:36.090516 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.090331    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:36.090516 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.090355    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:36.090516 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.090368    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:36.090516 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.090382    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/989c3294-e402-4c4d-8369-ca96d89fa5cd-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:36.090516 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.090396    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:36.091917 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.091890    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-token" pod "989c3294-e402-4c4d-8369-ca96d89fa5cd" (UID: "989c3294-e402-4c4d-8369-ca96d89fa5cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:36.093585 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.093388    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-podinfo" pod "989c3294-e402-4c4d-8369-ca96d89fa5cd" (UID: "989c3294-e402-4c4d-8369-ca96d89fa5cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:37:36.093766 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.093733    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-envoy" pod "989c3294-e402-4c4d-8369-ca96d89fa5cd" (UID: "989c3294-e402-4c4d-8369-ca96d89fa5cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:36.094923 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.094892    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/989c3294-e402-4c4d-8369-ca96d89fa5cd-kube-api-access-2gl7g" pod "989c3294-e402-4c4d-8369-ca96d89fa5cd" (UID: "989c3294-e402-4c4d-8369-ca96d89fa5cd"). InnerVolumeSpecName "kube-api-access-2gl7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:36.192663 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.192625    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:36.192663 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.192661    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:36.192868 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.192676    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/989c3294-e402-4c4d-8369-ca96d89fa5cd-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:36.192868 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.192695    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gl7g\" (UniqueName: \"kubernetes.io/projected/989c3294-e402-4c4d-8369-ca96d89fa5cd-kube-api-access-2gl7g\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:36.220667 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.220634    4421 generic.go:356] "Generic (PLEG): container finished" podID="989c3294-e402-4c4d-8369-ca96d89fa5cd" containerID="21eac988b1a95ccb158ef275484c9b99b715b60fbb79d72d5c59e8004e70ce71" exitCode=0
Nov 03 20:37:36.220850 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.220709    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"
Nov 03 20:37:36.220850 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.220736    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5" event={"ID":"989c3294-e402-4c4d-8369-ca96d89fa5cd","Type":"ContainerDied","Data":"21eac988b1a95ccb158ef275484c9b99b715b60fbb79d72d5c59e8004e70ce71"}
Nov 03 20:37:36.220850 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.220807    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5" event={"ID":"989c3294-e402-4c4d-8369-ca96d89fa5cd","Type":"ContainerDied","Data":"81f904131f3e83befbd19954bff673900f4b3ced173bbae407c783d38165fe1a"}
Nov 03 20:37:36.220850 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.220839    4421 scope.go:122] "RemoveContainer" containerID="21eac988b1a95ccb158ef275484c9b99b715b60fbb79d72d5c59e8004e70ce71"
Nov 03 20:37:36.236924 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.236903    4421 scope.go:122] "RemoveContainer" containerID="21eac988b1a95ccb158ef275484c9b99b715b60fbb79d72d5c59e8004e70ce71"
Nov 03 20:37:36.249692 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:37:36.237649    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21eac988b1a95ccb158ef275484c9b99b715b60fbb79d72d5c59e8004e70ce71\": container with ID starting with 21eac988b1a95ccb158ef275484c9b99b715b60fbb79d72d5c59e8004e70ce71 not found: ID does not exist" containerID="21eac988b1a95ccb158ef275484c9b99b715b60fbb79d72d5c59e8004e70ce71"
Nov 03 20:37:36.249692 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.237700    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21eac988b1a95ccb158ef275484c9b99b715b60fbb79d72d5c59e8004e70ce71"} err="failed to get container status \"21eac988b1a95ccb158ef275484c9b99b715b60fbb79d72d5c59e8004e70ce71\": rpc error: code = NotFound desc = could not find container \"21eac988b1a95ccb158ef275484c9b99b715b60fbb79d72d5c59e8004e70ce71\": container with ID starting with 21eac988b1a95ccb158ef275484c9b99b715b60fbb79d72d5c59e8004e70ce71 not found: ID does not exist"
Nov 03 20:37:36.266890 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.266859    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"]
Nov 03 20:37:36.276157 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:36.276124    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-p2bj5"]
Nov 03 20:37:37.939438 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:37:37.939410    4421 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a14b4da_8ce7_457e_89b8_6e4788d924f0.slice/crio-ec9d322207b181bfaba3248655446b57cc3138cd2dd31cd41e0cbe3b529d6eff.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:37:38.028960 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.028925    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="989c3294-e402-4c4d-8369-ca96d89fa5cd" path="/var/lib/kubelet/pods/989c3294-e402-4c4d-8369-ca96d89fa5cd/volumes"
Nov 03 20:37:38.243301 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.243273    4421 generic.go:356] "Generic (PLEG): container finished" podID="e840e124-446a-4ef7-8ed3-abc704048d25" containerID="a545fe0897cffeef80b749e593bc1323b971d280fda0d0709373c4ceaa0bd31c" exitCode=0
Nov 03 20:37:38.243477 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.243403    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw" event={"ID":"e840e124-446a-4ef7-8ed3-abc704048d25","Type":"ContainerDied","Data":"a545fe0897cffeef80b749e593bc1323b971d280fda0d0709373c4ceaa0bd31c"}
Nov 03 20:37:38.245343 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.245298    4421 generic.go:356] "Generic (PLEG): container finished" podID="5a14b4da-8ce7-457e-89b8-6e4788d924f0" containerID="ec9d322207b181bfaba3248655446b57cc3138cd2dd31cd41e0cbe3b529d6eff" exitCode=0
Nov 03 20:37:38.245517 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.245455    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt" event={"ID":"5a14b4da-8ce7-457e-89b8-6e4788d924f0","Type":"ContainerDied","Data":"ec9d322207b181bfaba3248655446b57cc3138cd2dd31cd41e0cbe3b529d6eff"}
Nov 03 20:37:38.250018 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.247742    4421 generic.go:356] "Generic (PLEG): container finished" podID="3844071e-3d45-455b-91be-3de079eacafa" containerID="bf94f44daf6fa955fb98218662473ead8be0652518b77894f77d37dff817a3b1" exitCode=0
Nov 03 20:37:38.250018 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.247813    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b" event={"ID":"3844071e-3d45-455b-91be-3de079eacafa","Type":"ContainerDied","Data":"bf94f44daf6fa955fb98218662473ead8be0652518b77894f77d37dff817a3b1"}
Nov 03 20:37:38.250018 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.249882    4421 generic.go:356] "Generic (PLEG): container finished" podID="3d2b0564-e201-4bab-9dec-422ecc7955b1" containerID="ac70a7a5a44774198c2eaf131db35305d9d0f77dfefb7ebf50f57a271607ab65" exitCode=0
Nov 03 20:37:38.250018 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.250013    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4" event={"ID":"3d2b0564-e201-4bab-9dec-422ecc7955b1","Type":"ContainerDied","Data":"ac70a7a5a44774198c2eaf131db35305d9d0f77dfefb7ebf50f57a271607ab65"}
Nov 03 20:37:38.281189 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.281159    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:37:38.305872 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.305758    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e840e124-446a-4ef7-8ed3-abc704048d25-kube-api-access-q6tjw\" (UniqueName: \"kubernetes.io/projected/e840e124-446a-4ef7-8ed3-abc704048d25-kube-api-access-q6tjw\") pod \"e840e124-446a-4ef7-8ed3-abc704048d25\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") "
Nov 03 20:37:38.306277 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.306225    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-istio-data\") pod \"e840e124-446a-4ef7-8ed3-abc704048d25\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") "
Nov 03 20:37:38.306618 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.306491    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e840e124-446a-4ef7-8ed3-abc704048d25-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e840e124-446a-4ef7-8ed3-abc704048d25-istiod-ca-cert\") pod \"e840e124-446a-4ef7-8ed3-abc704048d25\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") "
Nov 03 20:37:38.306942 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.306878    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-workload-certs\") pod \"e840e124-446a-4ef7-8ed3-abc704048d25\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") "
Nov 03 20:37:38.307259 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.307153    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e840e124-446a-4ef7-8ed3-abc704048d25-istio-token\" (UniqueName: \"kubernetes.io/projected/e840e124-446a-4ef7-8ed3-abc704048d25-istio-token\") pod \"e840e124-446a-4ef7-8ed3-abc704048d25\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") "
Nov 03 20:37:38.307259 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.307229    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e840e124-446a-4ef7-8ed3-abc704048d25-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e840e124-446a-4ef7-8ed3-abc704048d25-istio-podinfo\") pod \"e840e124-446a-4ef7-8ed3-abc704048d25\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") "
Nov 03 20:37:38.307721 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.307682    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-workload-socket\") pod \"e840e124-446a-4ef7-8ed3-abc704048d25\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") "
Nov 03 20:37:38.308041 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.307975    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-credential-socket\") pod \"e840e124-446a-4ef7-8ed3-abc704048d25\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") "
Nov 03 20:37:38.308281 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.308182    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-istio-envoy\") pod \"e840e124-446a-4ef7-8ed3-abc704048d25\" (UID: \"e840e124-446a-4ef7-8ed3-abc704048d25\") "
Nov 03 20:37:38.320777 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.320738    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-istio-envoy" pod "e840e124-446a-4ef7-8ed3-abc704048d25" (UID: "e840e124-446a-4ef7-8ed3-abc704048d25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:38.330123 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.330057    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e840e124-446a-4ef7-8ed3-abc704048d25-kube-api-access-q6tjw" pod "e840e124-446a-4ef7-8ed3-abc704048d25" (UID: "e840e124-446a-4ef7-8ed3-abc704048d25"). InnerVolumeSpecName "kube-api-access-q6tjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:38.338780 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.338748    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-workload-socket" pod "e840e124-446a-4ef7-8ed3-abc704048d25" (UID: "e840e124-446a-4ef7-8ed3-abc704048d25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:38.339655 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.339332    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-istio-data" pod "e840e124-446a-4ef7-8ed3-abc704048d25" (UID: "e840e124-446a-4ef7-8ed3-abc704048d25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:38.339655 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.339368    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e840e124-446a-4ef7-8ed3-abc704048d25-istiod-ca-cert" pod "e840e124-446a-4ef7-8ed3-abc704048d25" (UID: "e840e124-446a-4ef7-8ed3-abc704048d25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:38.339655 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.339549    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-credential-socket" pod "e840e124-446a-4ef7-8ed3-abc704048d25" (UID: "e840e124-446a-4ef7-8ed3-abc704048d25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:38.339655 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.339642    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-workload-certs" pod "e840e124-446a-4ef7-8ed3-abc704048d25" (UID: "e840e124-446a-4ef7-8ed3-abc704048d25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:38.343915 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.343892    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e840e124-446a-4ef7-8ed3-abc704048d25-istio-token" pod "e840e124-446a-4ef7-8ed3-abc704048d25" (UID: "e840e124-446a-4ef7-8ed3-abc704048d25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:38.354583 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.354551    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e840e124-446a-4ef7-8ed3-abc704048d25-istio-podinfo" pod "e840e124-446a-4ef7-8ed3-abc704048d25" (UID: "e840e124-446a-4ef7-8ed3-abc704048d25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:37:38.366450 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.366396    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:37:38.370867 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.370795    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:37:38.376009 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.375991    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:37:38.409926 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.409898    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d2b0564-e201-4bab-9dec-422ecc7955b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2b0564-e201-4bab-9dec-422ecc7955b1-istiod-ca-cert\") pod \"3d2b0564-e201-4bab-9dec-422ecc7955b1\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") "
Nov 03 20:37:38.409926 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.409936    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-envoy\") pod \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") "
Nov 03 20:37:38.410188 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.409964    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-workload-certs\") pod \"3d2b0564-e201-4bab-9dec-422ecc7955b1\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") "
Nov 03 20:37:38.410188 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.410003    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-istio-envoy\") pod \"3844071e-3d45-455b-91be-3de079eacafa\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") "
Nov 03 20:37:38.410188 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.410072    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-token\" (UniqueName: \"kubernetes.io/projected/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-token\") pod \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") "
Nov 03 20:37:38.410188 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.410101    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-data\") pod \"3d2b0564-e201-4bab-9dec-422ecc7955b1\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") "
Nov 03 20:37:38.410188 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.410137    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3844071e-3d45-455b-91be-3de079eacafa-istio-token\" (UniqueName: \"kubernetes.io/projected/3844071e-3d45-455b-91be-3de079eacafa-istio-token\") pod \"3844071e-3d45-455b-91be-3de079eacafa\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") "
Nov 03 20:37:38.410188 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.410154    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-istio-data\") pod \"3844071e-3d45-455b-91be-3de079eacafa\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") "
Nov 03 20:37:38.410188 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.410185    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d2b0564-e201-4bab-9dec-422ecc7955b1-kube-api-access-txswl\" (UniqueName: \"kubernetes.io/projected/3d2b0564-e201-4bab-9dec-422ecc7955b1-kube-api-access-txswl\") pod \"3d2b0564-e201-4bab-9dec-422ecc7955b1\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") "
Nov 03 20:37:38.410579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.410213    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3844071e-3d45-455b-91be-3de079eacafa-kube-api-access-nq226\" (UniqueName: \"kubernetes.io/projected/3844071e-3d45-455b-91be-3de079eacafa-kube-api-access-nq226\") pod \"3844071e-3d45-455b-91be-3de079eacafa\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") "
Nov 03 20:37:38.410579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.410241    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-data\") pod \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") "
Nov 03 20:37:38.410579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.410278    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-credential-socket\") pod \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") "
Nov 03 20:37:38.410579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.410343    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-workload-socket\") pod \"3844071e-3d45-455b-91be-3de079eacafa\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") "
Nov 03 20:37:38.410579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.410379    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-workload-socket\") pod \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") "
Nov 03 20:37:38.410579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.410414    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-podinfo\") pod \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") "
Nov 03 20:37:38.410579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.410442    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istiod-ca-cert\") pod \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") "
Nov 03 20:37:38.410579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.410509    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-workload-socket\") pod \"3d2b0564-e201-4bab-9dec-422ecc7955b1\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") "
Nov 03 20:37:38.410579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.410553    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-envoy\") pod \"3d2b0564-e201-4bab-9dec-422ecc7955b1\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") "
Nov 03 20:37:38.411081 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.410590    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-workload-certs\") pod \"3844071e-3d45-455b-91be-3de079eacafa\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") "
Nov 03 20:37:38.411081 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.410659    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-token\" (UniqueName: \"kubernetes.io/projected/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-token\") pod \"3d2b0564-e201-4bab-9dec-422ecc7955b1\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") "
Nov 03 20:37:38.411081 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.410754    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-credential-socket\") pod \"3844071e-3d45-455b-91be-3de079eacafa\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") "
Nov 03 20:37:38.411081 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.410807    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3844071e-3d45-455b-91be-3de079eacafa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3844071e-3d45-455b-91be-3de079eacafa-istio-podinfo\") pod \"3844071e-3d45-455b-91be-3de079eacafa\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") "
Nov 03 20:37:38.411081 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.410846    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-credential-socket\") pod \"3d2b0564-e201-4bab-9dec-422ecc7955b1\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") "
Nov 03 20:37:38.411081 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.410873    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-podinfo\") pod \"3d2b0564-e201-4bab-9dec-422ecc7955b1\" (UID: \"3d2b0564-e201-4bab-9dec-422ecc7955b1\") "
Nov 03 20:37:38.411081 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.410948    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a14b4da-8ce7-457e-89b8-6e4788d924f0-kube-api-access-sbxqb\" (UniqueName: \"kubernetes.io/projected/5a14b4da-8ce7-457e-89b8-6e4788d924f0-kube-api-access-sbxqb\") pod \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") "
Nov 03 20:37:38.411081 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.410983    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-workload-certs\") pod \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\" (UID: \"5a14b4da-8ce7-457e-89b8-6e4788d924f0\") "
Nov 03 20:37:38.411081 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.411017    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3844071e-3d45-455b-91be-3de079eacafa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3844071e-3d45-455b-91be-3de079eacafa-istiod-ca-cert\") pod \"3844071e-3d45-455b-91be-3de079eacafa\" (UID: \"3844071e-3d45-455b-91be-3de079eacafa\") "
Nov 03 20:37:38.411888 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.411805    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.411888 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.411830    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6tjw\" (UniqueName: \"kubernetes.io/projected/e840e124-446a-4ef7-8ed3-abc704048d25-kube-api-access-q6tjw\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.411888 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.411846    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.412787 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.411917    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-workload-socket" pod "3844071e-3d45-455b-91be-3de079eacafa" (UID: "3844071e-3d45-455b-91be-3de079eacafa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:38.413181 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.413080    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-credential-socket" pod "3d2b0564-e201-4bab-9dec-422ecc7955b1" (UID: "3d2b0564-e201-4bab-9dec-422ecc7955b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:38.413181 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.413096    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-workload-certs" pod "3d2b0564-e201-4bab-9dec-422ecc7955b1" (UID: "3d2b0564-e201-4bab-9dec-422ecc7955b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:38.413355 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.413292    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-credential-socket" pod "3844071e-3d45-455b-91be-3de079eacafa" (UID: "3844071e-3d45-455b-91be-3de079eacafa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:38.413723 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.413433    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e840e124-446a-4ef7-8ed3-abc704048d25-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.413723 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.413461    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.413723 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.413477    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e840e124-446a-4ef7-8ed3-abc704048d25-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.413723 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.413490    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e840e124-446a-4ef7-8ed3-abc704048d25-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.413723 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.413506    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.413723 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.413520    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e840e124-446a-4ef7-8ed3-abc704048d25-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.413723 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.413436    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2b0564-e201-4bab-9dec-422ecc7955b1-istiod-ca-cert" pod "3d2b0564-e201-4bab-9dec-422ecc7955b1" (UID: "3d2b0564-e201-4bab-9dec-422ecc7955b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:38.413723 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.413529    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-data" pod "3d2b0564-e201-4bab-9dec-422ecc7955b1" (UID: "3d2b0564-e201-4bab-9dec-422ecc7955b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:38.414207 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.413769    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-istio-data" pod "3844071e-3d45-455b-91be-3de079eacafa" (UID: "3844071e-3d45-455b-91be-3de079eacafa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:38.414363 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.414220    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-data" pod "5a14b4da-8ce7-457e-89b8-6e4788d924f0" (UID: "5a14b4da-8ce7-457e-89b8-6e4788d924f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:38.415285 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.414937    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istiod-ca-cert" pod "5a14b4da-8ce7-457e-89b8-6e4788d924f0" (UID: "5a14b4da-8ce7-457e-89b8-6e4788d924f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:38.415692 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.415412    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-workload-socket" pod "3d2b0564-e201-4bab-9dec-422ecc7955b1" (UID: "3d2b0564-e201-4bab-9dec-422ecc7955b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:38.416109 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.416082    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-workload-certs" pod "3844071e-3d45-455b-91be-3de079eacafa" (UID: "3844071e-3d45-455b-91be-3de079eacafa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:38.416202 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.416152    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-credential-socket" pod "5a14b4da-8ce7-457e-89b8-6e4788d924f0" (UID: "5a14b4da-8ce7-457e-89b8-6e4788d924f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:38.416202 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.416192    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-workload-socket" pod "5a14b4da-8ce7-457e-89b8-6e4788d924f0" (UID: "5a14b4da-8ce7-457e-89b8-6e4788d924f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:38.416432 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.416409    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-workload-certs" pod "5a14b4da-8ce7-457e-89b8-6e4788d924f0" (UID: "5a14b4da-8ce7-457e-89b8-6e4788d924f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:38.416866 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.416805    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3844071e-3d45-455b-91be-3de079eacafa-istiod-ca-cert" pod "3844071e-3d45-455b-91be-3de079eacafa" (UID: "3844071e-3d45-455b-91be-3de079eacafa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:38.417737 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.417699    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-podinfo" pod "3d2b0564-e201-4bab-9dec-422ecc7955b1" (UID: "3d2b0564-e201-4bab-9dec-422ecc7955b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:37:38.425985 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.425940    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-token" pod "5a14b4da-8ce7-457e-89b8-6e4788d924f0" (UID: "5a14b4da-8ce7-457e-89b8-6e4788d924f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:38.425985 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.425950    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-istio-envoy" pod "3844071e-3d45-455b-91be-3de079eacafa" (UID: "3844071e-3d45-455b-91be-3de079eacafa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:38.425985 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.425979    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2b0564-e201-4bab-9dec-422ecc7955b1-kube-api-access-txswl" pod "3d2b0564-e201-4bab-9dec-422ecc7955b1" (UID: "3d2b0564-e201-4bab-9dec-422ecc7955b1"). InnerVolumeSpecName "kube-api-access-txswl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:38.426209 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.425997    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-envoy" pod "5a14b4da-8ce7-457e-89b8-6e4788d924f0" (UID: "5a14b4da-8ce7-457e-89b8-6e4788d924f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:38.426265 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.426230    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3844071e-3d45-455b-91be-3de079eacafa-kube-api-access-nq226" pod "3844071e-3d45-455b-91be-3de079eacafa" (UID: "3844071e-3d45-455b-91be-3de079eacafa"). InnerVolumeSpecName "kube-api-access-nq226". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:38.426512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.426412    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3844071e-3d45-455b-91be-3de079eacafa-istio-token" pod "3844071e-3d45-455b-91be-3de079eacafa" (UID: "3844071e-3d45-455b-91be-3de079eacafa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:38.426512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.426415    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3844071e-3d45-455b-91be-3de079eacafa-istio-podinfo" pod "3844071e-3d45-455b-91be-3de079eacafa" (UID: "3844071e-3d45-455b-91be-3de079eacafa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:37:38.426512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.426484    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-token" pod "3d2b0564-e201-4bab-9dec-422ecc7955b1" (UID: "3d2b0564-e201-4bab-9dec-422ecc7955b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:38.428636 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.428582    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-podinfo" pod "5a14b4da-8ce7-457e-89b8-6e4788d924f0" (UID: "5a14b4da-8ce7-457e-89b8-6e4788d924f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:37:38.432490 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.432460    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-envoy" pod "3d2b0564-e201-4bab-9dec-422ecc7955b1" (UID: "3d2b0564-e201-4bab-9dec-422ecc7955b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:38.439850 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.433222    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a14b4da-8ce7-457e-89b8-6e4788d924f0-kube-api-access-sbxqb" pod "5a14b4da-8ce7-457e-89b8-6e4788d924f0" (UID: "5a14b4da-8ce7-457e-89b8-6e4788d924f0"). InnerVolumeSpecName "kube-api-access-sbxqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:38.515402 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.515359    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.515402 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.515396    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3844071e-3d45-455b-91be-3de079eacafa-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.515402 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.515411    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.515742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.515423    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.515742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.515463    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbxqb\" (UniqueName: \"kubernetes.io/projected/5a14b4da-8ce7-457e-89b8-6e4788d924f0-kube-api-access-sbxqb\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.515742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.515480    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.515742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.515492    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3844071e-3d45-455b-91be-3de079eacafa-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.515742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.515503    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2b0564-e201-4bab-9dec-422ecc7955b1-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.515742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.515516    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.515742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.515528    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.515742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.515539    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.515742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.515554    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.515742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.515565    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.515742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.515577    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3844071e-3d45-455b-91be-3de079eacafa-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.515742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.515589    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.515742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.515620    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txswl\" (UniqueName: \"kubernetes.io/projected/3d2b0564-e201-4bab-9dec-422ecc7955b1-kube-api-access-txswl\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.515742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.515634    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq226\" (UniqueName: \"kubernetes.io/projected/3844071e-3d45-455b-91be-3de079eacafa-kube-api-access-nq226\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.515742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.515648    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.515742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.515663    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.515742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.515677    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.515742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.515691    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a14b4da-8ce7-457e-89b8-6e4788d924f0-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.515742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.515705    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.515742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.515720    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a14b4da-8ce7-457e-89b8-6e4788d924f0-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.515742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.515733    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.515742 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.515746    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.516561 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.515759    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3844071e-3d45-455b-91be-3de079eacafa-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:38.516561 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:38.515772    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2b0564-e201-4bab-9dec-422ecc7955b1-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:39.259108 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:39.259078    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b" event={"ID":"3844071e-3d45-455b-91be-3de079eacafa","Type":"ContainerDied","Data":"5667df3bb0ba35e584738213a561ae10f7c75536977a00e4ec47b01056cee24e"}
Nov 03 20:37:39.259538 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:39.259139    4421 scope.go:122] "RemoveContainer" containerID="bf94f44daf6fa955fb98218662473ead8be0652518b77894f77d37dff817a3b1"
Nov 03 20:37:39.259538 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:39.259089    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"
Nov 03 20:37:39.260925 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:39.260890    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4" event={"ID":"3d2b0564-e201-4bab-9dec-422ecc7955b1","Type":"ContainerDied","Data":"d9c68783fd4b11d78f2e0b3cb415d08cd6781f168e7e82d2058921e16112c4ed"}
Nov 03 20:37:39.261047 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:39.261029    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"
Nov 03 20:37:39.268301 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:39.268283    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"
Nov 03 20:37:39.268432 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:39.268316    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw" event={"ID":"e840e124-446a-4ef7-8ed3-abc704048d25","Type":"ContainerDied","Data":"8bf9bf356ea121afe51f521c77d73d1acd1bf29909823dbf10d49cc24f80b91e"}
Nov 03 20:37:39.272380 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:39.272325    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"
Nov 03 20:37:39.272380 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:39.272333    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt" event={"ID":"5a14b4da-8ce7-457e-89b8-6e4788d924f0","Type":"ContainerDied","Data":"8e396394a94d1304afb2a5c38f5b7c304987fddaffcf3abf09493957728c2d83"}
Nov 03 20:37:39.273682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:39.273665    4421 scope.go:122] "RemoveContainer" containerID="ac70a7a5a44774198c2eaf131db35305d9d0f77dfefb7ebf50f57a271607ab65"
Nov 03 20:37:39.286906 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:39.286890    4421 scope.go:122] "RemoveContainer" containerID="a545fe0897cffeef80b749e593bc1323b971d280fda0d0709373c4ceaa0bd31c"
Nov 03 20:37:39.297283 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:39.297259    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"]
Nov 03 20:37:39.303369 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:39.303324    4421 scope.go:122] "RemoveContainer" containerID="ec9d322207b181bfaba3248655446b57cc3138cd2dd31cd41e0cbe3b529d6eff"
Nov 03 20:37:39.305181 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:39.305155    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-h898b"]
Nov 03 20:37:39.313851 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:39.313796    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"]
Nov 03 20:37:39.319633 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:39.319590    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-b4nvw"]
Nov 03 20:37:39.337992 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:39.337966    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"]
Nov 03 20:37:39.342803 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:39.342778    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-d9jk4"]
Nov 03 20:37:39.351623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:39.351586    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"]
Nov 03 20:37:39.356950 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:39.356930    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wgvpt"]
Nov 03 20:37:40.019101 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:40.019063    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3844071e-3d45-455b-91be-3de079eacafa" path="/var/lib/kubelet/pods/3844071e-3d45-455b-91be-3de079eacafa/volumes"
Nov 03 20:37:40.020021 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:40.019997    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d2b0564-e201-4bab-9dec-422ecc7955b1" path="/var/lib/kubelet/pods/3d2b0564-e201-4bab-9dec-422ecc7955b1/volumes"
Nov 03 20:37:40.020661 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:40.020642    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a14b4da-8ce7-457e-89b8-6e4788d924f0" path="/var/lib/kubelet/pods/5a14b4da-8ce7-457e-89b8-6e4788d924f0/volumes"
Nov 03 20:37:40.021585 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:40.021556    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e840e124-446a-4ef7-8ed3-abc704048d25" path="/var/lib/kubelet/pods/e840e124-446a-4ef7-8ed3-abc704048d25/volumes"
Nov 03 20:37:45.124098 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:37:45.124054    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:37:49.633264 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:37:49.633179    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:38:02.061578 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:02.061542    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz"]
Nov 03 20:38:07.062677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:07.062632    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz" podUID="4dee47b3-4df5-4b7b-bb29-c82ab58e2023" containerName="payload-processing" containerID="cri-o://5b9e137fca15aef527736a8a1645787dedcae6a06193bfe12244d9b1822ec6da" gracePeriod=25
Nov 03 20:38:07.363727 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:07.363704    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz"
Nov 03 20:38:07.583807 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:07.583717    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-provider-credentials\") pod \"4dee47b3-4df5-4b7b-bb29-c82ab58e2023\" (UID: \"4dee47b3-4df5-4b7b-bb29-c82ab58e2023\") "
Nov 03 20:38:07.583996 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:07.583893    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-kube-api-access-mrhsv\" (UniqueName: \"kubernetes.io/projected/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-kube-api-access-mrhsv\") pod \"4dee47b3-4df5-4b7b-bb29-c82ab58e2023\" (UID: \"4dee47b3-4df5-4b7b-bb29-c82ab58e2023\") "
Nov 03 20:38:07.584069 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:07.583996    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-routing-overlay-volume\") pod \"4dee47b3-4df5-4b7b-bb29-c82ab58e2023\" (UID: \"4dee47b3-4df5-4b7b-bb29-c82ab58e2023\") "
Nov 03 20:38:07.584198 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:07.584182    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-plugins-config-volume\") pod \"4dee47b3-4df5-4b7b-bb29-c82ab58e2023\" (UID: \"4dee47b3-4df5-4b7b-bb29-c82ab58e2023\") "
Nov 03 20:38:07.584437 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:07.584406    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-routing-overlay-volume" pod "4dee47b3-4df5-4b7b-bb29-c82ab58e2023" (UID: "4dee47b3-4df5-4b7b-bb29-c82ab58e2023"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:07.584520 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:07.584507    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-plugins-config-volume" pod "4dee47b3-4df5-4b7b-bb29-c82ab58e2023" (UID: "4dee47b3-4df5-4b7b-bb29-c82ab58e2023"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:07.584896 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:07.584878    4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-routing-overlay-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:07.584956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:07.584899    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:07.586193 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:07.586168    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-kube-api-access-mrhsv" pod "4dee47b3-4df5-4b7b-bb29-c82ab58e2023" (UID: "4dee47b3-4df5-4b7b-bb29-c82ab58e2023"). InnerVolumeSpecName "kube-api-access-mrhsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:07.586282 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:07.586201    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-provider-credentials" pod "4dee47b3-4df5-4b7b-bb29-c82ab58e2023" (UID: "4dee47b3-4df5-4b7b-bb29-c82ab58e2023"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:07.686579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:07.686548    4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-provider-credentials\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:07.686579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:07.686577    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrhsv\" (UniqueName: \"kubernetes.io/projected/4dee47b3-4df5-4b7b-bb29-c82ab58e2023-kube-api-access-mrhsv\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:07.818819 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:07.818788    4421 generic.go:356] "Generic (PLEG): container finished" podID="4dee47b3-4df5-4b7b-bb29-c82ab58e2023" containerID="5b9e137fca15aef527736a8a1645787dedcae6a06193bfe12244d9b1822ec6da" exitCode=0
Nov 03 20:38:07.818994 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:07.818835    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz"
Nov 03 20:38:07.818994 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:07.818868    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz" event={"ID":"4dee47b3-4df5-4b7b-bb29-c82ab58e2023","Type":"ContainerDied","Data":"5b9e137fca15aef527736a8a1645787dedcae6a06193bfe12244d9b1822ec6da"}
Nov 03 20:38:07.818994 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:07.818922    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz" event={"ID":"4dee47b3-4df5-4b7b-bb29-c82ab58e2023","Type":"ContainerDied","Data":"d0389b1e15639096acff4eae3f78f4386e6f83a83561a443a8e42cab7f0662d4"}
Nov 03 20:38:07.818994 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:07.818943    4421 scope.go:122] "RemoveContainer" containerID="5b9e137fca15aef527736a8a1645787dedcae6a06193bfe12244d9b1822ec6da"
Nov 03 20:38:07.831409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:07.831391    4421 scope.go:122] "RemoveContainer" containerID="5b9e137fca15aef527736a8a1645787dedcae6a06193bfe12244d9b1822ec6da"
Nov 03 20:38:07.831720 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:38:07.831699    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b9e137fca15aef527736a8a1645787dedcae6a06193bfe12244d9b1822ec6da\": container with ID starting with 5b9e137fca15aef527736a8a1645787dedcae6a06193bfe12244d9b1822ec6da not found: ID does not exist" containerID="5b9e137fca15aef527736a8a1645787dedcae6a06193bfe12244d9b1822ec6da"
Nov 03 20:38:07.831835 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:07.831733    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b9e137fca15aef527736a8a1645787dedcae6a06193bfe12244d9b1822ec6da"} err="failed to get container status \"5b9e137fca15aef527736a8a1645787dedcae6a06193bfe12244d9b1822ec6da\": rpc error: code = NotFound desc = could not find container \"5b9e137fca15aef527736a8a1645787dedcae6a06193bfe12244d9b1822ec6da\": container with ID starting with 5b9e137fca15aef527736a8a1645787dedcae6a06193bfe12244d9b1822ec6da not found: ID does not exist"
Nov 03 20:38:07.844268 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:07.844215    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz"]
Nov 03 20:38:07.848404 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:07.848383    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6350b9/payload-processing-external-model-e2e-extmod-b-6350b9-598dbrzmz"]
Nov 03 20:38:08.018869 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:08.018835    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dee47b3-4df5-4b7b-bb29-c82ab58e2023" path="/var/lib/kubelet/pods/4dee47b3-4df5-4b7b-bb29-c82ab58e2023/volumes"
Nov 03 20:38:19.631023 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:38:19.630985    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:38:30.377299 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.377265    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"]
Nov 03 20:38:30.378349 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.378324    4421 cpu_manager.go:414] "containerMap: removing container" podUID="3d2b0564-e201-4bab-9dec-422ecc7955b1" containerName="istio-proxy"
Nov 03 20:38:30.378349 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.378348    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d2b0564-e201-4bab-9dec-422ecc7955b1" containerName="istio-proxy"
Nov 03 20:38:30.378542 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.378376    4421 cpu_manager.go:414] "containerMap: removing container" podUID="3844071e-3d45-455b-91be-3de079eacafa" containerName="istio-proxy"
Nov 03 20:38:30.378542 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.378383    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3844071e-3d45-455b-91be-3de079eacafa" containerName="istio-proxy"
Nov 03 20:38:30.378542 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.378400    4421 cpu_manager.go:414] "containerMap: removing container" podUID="e840e124-446a-4ef7-8ed3-abc704048d25" containerName="istio-proxy"
Nov 03 20:38:30.378542 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.378406    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e840e124-446a-4ef7-8ed3-abc704048d25" containerName="istio-proxy"
Nov 03 20:38:30.378542 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.378418    4421 cpu_manager.go:414] "containerMap: removing container" podUID="5a14b4da-8ce7-457e-89b8-6e4788d924f0" containerName="istio-proxy"
Nov 03 20:38:30.378542 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.378424    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a14b4da-8ce7-457e-89b8-6e4788d924f0" containerName="istio-proxy"
Nov 03 20:38:30.378542 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.378476    4421 cpu_manager.go:414] "containerMap: removing container" podUID="4dee47b3-4df5-4b7b-bb29-c82ab58e2023" containerName="payload-processing"
Nov 03 20:38:30.378542 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.378485    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dee47b3-4df5-4b7b-bb29-c82ab58e2023" containerName="payload-processing"
Nov 03 20:38:30.378542 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.378515    4421 cpu_manager.go:414] "containerMap: removing container" podUID="8d1d77d9-42e6-4eac-a09b-4cad11a204ba" containerName="payload-processing"
Nov 03 20:38:30.378542 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.378522    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d1d77d9-42e6-4eac-a09b-4cad11a204ba" containerName="payload-processing"
Nov 03 20:38:30.378542 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.378534    4421 cpu_manager.go:414] "containerMap: removing container" podUID="989c3294-e402-4c4d-8369-ca96d89fa5cd" containerName="istio-proxy"
Nov 03 20:38:30.378542 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.378541    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="989c3294-e402-4c4d-8369-ca96d89fa5cd" containerName="istio-proxy"
Nov 03 20:38:30.379211 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.378766    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dee47b3-4df5-4b7b-bb29-c82ab58e2023" containerName="payload-processing"
Nov 03 20:38:30.379211 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.378785    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d1d77d9-42e6-4eac-a09b-4cad11a204ba" containerName="payload-processing"
Nov 03 20:38:30.379211 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.378795    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="989c3294-e402-4c4d-8369-ca96d89fa5cd" containerName="istio-proxy"
Nov 03 20:38:30.379211 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.378824    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d2b0564-e201-4bab-9dec-422ecc7955b1" containerName="istio-proxy"
Nov 03 20:38:30.379211 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.378835    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="3844071e-3d45-455b-91be-3de079eacafa" containerName="istio-proxy"
Nov 03 20:38:30.379211 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.378843    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e840e124-446a-4ef7-8ed3-abc704048d25" containerName="istio-proxy"
Nov 03 20:38:30.379211 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.378853    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a14b4da-8ce7-457e-89b8-6e4788d924f0" containerName="istio-proxy"
Nov 03 20:38:30.385264 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.385015    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.389370 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.389344    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-37aa95-openshift-default-dockercfg-6h6k6\""
Nov 03 20:38:30.390336 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.390312    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"]
Nov 03 20:38:30.527951 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.527909    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-workload-socket\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.528156 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.528060    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-token\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.528156 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.528105    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istiod-ca-cert\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.528268 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.528171    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-podinfo\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.528330 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.528315    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kjb4\" (UniqueName: \"kubernetes.io/projected/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-kube-api-access-7kjb4\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.528395 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.528382    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-envoy\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.528504 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.528488    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-workload-certs\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.528581 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.528565    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-credential-socket\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.528700 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.528668    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-data\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.630878 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.630789    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-envoy\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.631183 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.631162    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-workload-certs\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.631401 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.631382    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-credential-socket\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.633004 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.631513    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-workload-certs\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.633004 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.631569    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-data\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.633004 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.631681    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-workload-socket\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.633004 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.631790    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-token\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.633004 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.631837    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istiod-ca-cert\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.633004 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.631856    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-credential-socket\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.633004 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.631864    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-podinfo\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.633004 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.632039    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kjb4\" (UniqueName: \"kubernetes.io/projected/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-kube-api-access-7kjb4\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.633004 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.632090    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-workload-socket\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.633004 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.632458    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-data\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.634068 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.633125    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istiod-ca-cert\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.636260 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.636234    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-podinfo\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.636415 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.636386    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-envoy\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.674623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.674172    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-token\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.675647 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.675591    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kjb4\" (UniqueName: \"kubernetes.io/projected/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-kube-api-access-7kjb4\") pod \"e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") " pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:30.702745 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:30.702717    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:31.021524 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:38:31.021492    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5d4c6c_d6ce_4a10_b875_6016aaf24e65.slice/crio-a0d7ea33af6e5d525eac87778417e0ac04afa59130a69f115374ecdd0befa439 WatchSource:0}: Error finding container a0d7ea33af6e5d525eac87778417e0ac04afa59130a69f115374ecdd0befa439: Status 404 returned error can't find the container with id a0d7ea33af6e5d525eac87778417e0ac04afa59130a69f115374ecdd0befa439
Nov 03 20:38:31.021743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:31.021710    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"]
Nov 03 20:38:31.024866 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:31.024829    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:38:31.024955 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:31.024919    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:38:31.025024 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:31.024953    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:38:31.284164 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:31.284076    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn" event={"ID":"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65","Type":"ContainerStarted","Data":"dc022417d717572f02b905d78c3c77a7978d66cced992ce85aaddb5f1992d3a1"}
Nov 03 20:38:31.284164 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:31.284130    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn" event={"ID":"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65","Type":"ContainerStarted","Data":"a0d7ea33af6e5d525eac87778417e0ac04afa59130a69f115374ecdd0befa439"}
Nov 03 20:38:31.312076 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:31.312025    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn" podStartSLOduration=1.312011766 podStartE2EDuration="1.312011766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:38:31.31003852 +0000 UTC m=+2241.443166984" watchObservedRunningTime="2026-11-03 20:38:31.312011766 +0000 UTC m=+2241.445140228"
Nov 03 20:38:31.703316 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:31.703282    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:32.705436 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:32.705395    4421 patch_prober.go:29] interesting pod/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 03 20:38:32.705953 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:32.705472    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn" podUID="1e5d4c6c-d6ce-4a10-b875-6016aaf24e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 03 20:38:33.703036 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:33.702999    4421 patch_prober.go:29] interesting pod/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 03 20:38:33.703242 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:33.703068    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn" podUID="1e5d4c6c-d6ce-4a10-b875-6016aaf24e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 03 20:38:34.703044 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:34.703011    4421 patch_prober.go:29] interesting pod/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 03 20:38:34.703479 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:34.703065    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn" podUID="1e5d4c6c-d6ce-4a10-b875-6016aaf24e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 03 20:38:35.703185 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:35.703152    4421 patch_prober.go:29] interesting pod/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 03 20:38:35.703849 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:35.703199    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn" podUID="1e5d4c6c-d6ce-4a10-b875-6016aaf24e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 03 20:38:36.703021 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:36.702987    4421 patch_prober.go:29] interesting pod/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 03 20:38:36.704247 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:36.703049    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn" podUID="1e5d4c6c-d6ce-4a10-b875-6016aaf24e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 03 20:38:37.466678 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:37.466640    4421 generic.go:356] "Generic (PLEG): container finished" podID="0a2fda22-a9be-4bf5-8b2b-0e994221c92b" containerID="6138acb76efaa00253d360733024c7f31ec60285410bbb53cd396b7f98c0917a" exitCode=0
Nov 03 20:38:37.466894 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:37.466691    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j" event={"ID":"0a2fda22-a9be-4bf5-8b2b-0e994221c92b","Type":"ContainerDied","Data":"6138acb76efaa00253d360733024c7f31ec60285410bbb53cd396b7f98c0917a"}
Nov 03 20:38:37.467263 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:37.467240    4421 scope.go:122] "RemoveContainer" containerID="6138acb76efaa00253d360733024c7f31ec60285410bbb53cd396b7f98c0917a"
Nov 03 20:38:37.469324 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:37.469291    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:38:37.469445 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:37.469365    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:38:37.469445 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:37.469398    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:38:37.738199 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:37.738116    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:37.739779 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:37.739754    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:37.745234 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:37.743741    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:38:38.477961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:38.477909    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j" event={"ID":"0a2fda22-a9be-4bf5-8b2b-0e994221c92b","Type":"ContainerStarted","Data":"fc80028a097b5022a46019d0e712125499641aa2849f75f8367e210a39a7c4ba"}
Nov 03 20:38:39.017768 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:39.017719    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:38:39.019506 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:39.019478    4421 patch_prober.go:29] interesting pod/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 03 20:38:39.019662 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:39.019550    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j" podUID="0a2fda22-a9be-4bf5-8b2b-0e994221c92b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 03 20:38:39.697565 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:39.697536    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:38:40.018317 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:40.018223    4421 patch_prober.go:29] interesting pod/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 03 20:38:40.018317 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:40.018281    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j" podUID="0a2fda22-a9be-4bf5-8b2b-0e994221c92b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 03 20:38:41.017764 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:41.017723    4421 patch_prober.go:29] interesting pod/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 03 20:38:41.017956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:41.017790    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j" podUID="0a2fda22-a9be-4bf5-8b2b-0e994221c92b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 03 20:38:42.018227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:42.018193    4421 patch_prober.go:29] interesting pod/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 03 20:38:42.018655 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:42.018265    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j" podUID="0a2fda22-a9be-4bf5-8b2b-0e994221c92b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 03 20:38:43.022041 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:43.021999    4421 patch_prober.go:29] interesting pod/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 03 20:38:43.022913 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:43.022879    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j" podUID="0a2fda22-a9be-4bf5-8b2b-0e994221c92b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 03 20:38:44.017639 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:44.017588    4421 patch_prober.go:29] interesting pod/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 03 20:38:44.017821 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:44.017667    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j" podUID="0a2fda22-a9be-4bf5-8b2b-0e994221c92b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 03 20:38:45.022559 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:45.022518    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:38:45.023782 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:38:45.023760    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 20:38:49.629909 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:38:49.629872    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:39:19.638261 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:39:19.638167    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:39:23.746325 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:23.746295    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq"]
Nov 03 20:39:23.751729 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:23.751710    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq"
Nov 03 20:39:23.757672 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:23.757649    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"kube-root-ca.crt\""
Nov 03 20:39:23.757890 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:23.757769    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"openshift-service-ca.crt\""
Nov 03 20:39:23.758019 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:23.757926    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"routing-overlay\""
Nov 03 20:39:23.758310 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:23.758263    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"payload-processing-external-model-plugins-e2e-extmod-a-37aa95\""
Nov 03 20:39:23.758519 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:23.758302    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"payload-processing-external-model-e2e--0dce6837-dockercfg-c69fq\""
Nov 03 20:39:23.758519 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:23.758333    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"agc-cred-5797404fedc5\""
Nov 03 20:39:23.758519 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:23.758373    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"agc-cred-f1f89f088748\""
Nov 03 20:39:23.758926 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:23.758864    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"agc-cred-b599b816968c\""
Nov 03 20:39:23.765263 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:23.765240    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq"]
Nov 03 20:39:23.869333 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:23.869292    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/592d56a0-fb52-4715-b639-ca70e92763d6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq\" (UID: \"592d56a0-fb52-4715-b639-ca70e92763d6\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq"
Nov 03 20:39:23.869527 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:23.869411    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8llpr\" (UniqueName: \"kubernetes.io/projected/592d56a0-fb52-4715-b639-ca70e92763d6-kube-api-access-8llpr\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq\" (UID: \"592d56a0-fb52-4715-b639-ca70e92763d6\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq"
Nov 03 20:39:23.869527 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:23.869458    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/592d56a0-fb52-4715-b639-ca70e92763d6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq\" (UID: \"592d56a0-fb52-4715-b639-ca70e92763d6\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq"
Nov 03 20:39:23.869723 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:23.869551    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592d56a0-fb52-4715-b639-ca70e92763d6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq\" (UID: \"592d56a0-fb52-4715-b639-ca70e92763d6\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq"
Nov 03 20:39:23.971921 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:23.971876    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/592d56a0-fb52-4715-b639-ca70e92763d6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq\" (UID: \"592d56a0-fb52-4715-b639-ca70e92763d6\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq"
Nov 03 20:39:23.991310 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:23.991276    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8llpr\" (UniqueName: \"kubernetes.io/projected/592d56a0-fb52-4715-b639-ca70e92763d6-kube-api-access-8llpr\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq\" (UID: \"592d56a0-fb52-4715-b639-ca70e92763d6\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq"
Nov 03 20:39:23.992294 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:23.992265    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/592d56a0-fb52-4715-b639-ca70e92763d6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq\" (UID: \"592d56a0-fb52-4715-b639-ca70e92763d6\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq"
Nov 03 20:39:23.992520 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:23.992503    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592d56a0-fb52-4715-b639-ca70e92763d6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq\" (UID: \"592d56a0-fb52-4715-b639-ca70e92763d6\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq"
Nov 03 20:39:23.993901 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:23.993874    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592d56a0-fb52-4715-b639-ca70e92763d6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq\" (UID: \"592d56a0-fb52-4715-b639-ca70e92763d6\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq"
Nov 03 20:39:24.000283 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:24.000182    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/592d56a0-fb52-4715-b639-ca70e92763d6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq\" (UID: \"592d56a0-fb52-4715-b639-ca70e92763d6\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq"
Nov 03 20:39:24.023439 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:24.023406    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8llpr\" (UniqueName: \"kubernetes.io/projected/592d56a0-fb52-4715-b639-ca70e92763d6-kube-api-access-8llpr\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq\" (UID: \"592d56a0-fb52-4715-b639-ca70e92763d6\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq"
Nov 03 20:39:24.063442 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:24.063407    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/592d56a0-fb52-4715-b639-ca70e92763d6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq\" (UID: \"592d56a0-fb52-4715-b639-ca70e92763d6\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq"
Nov 03 20:39:24.110790 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:24.110756    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq"
Nov 03 20:39:24.510696 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:39:24.510667    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod592d56a0_fb52_4715_b639_ca70e92763d6.slice/crio-0145af07853bf495390c6d2a222b007fa491c7e56d4b67eec7b962bfe82696c2 WatchSource:0}: Error finding container 0145af07853bf495390c6d2a222b007fa491c7e56d4b67eec7b962bfe82696c2: Status 404 returned error can't find the container with id 0145af07853bf495390c6d2a222b007fa491c7e56d4b67eec7b962bfe82696c2
Nov 03 20:39:24.511136 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:24.511112    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq"]
Nov 03 20:39:25.225385 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:25.225351    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq" event={"ID":"592d56a0-fb52-4715-b639-ca70e92763d6","Type":"ContainerStarted","Data":"a6310627e967f618add0c135c96badf2e50353f91e06aed2ae8c2065e72c7716"}
Nov 03 20:39:25.225883 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:25.225413    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq"
Nov 03 20:39:25.225883 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:25.225432    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq" event={"ID":"592d56a0-fb52-4715-b639-ca70e92763d6","Type":"ContainerStarted","Data":"0145af07853bf495390c6d2a222b007fa491c7e56d4b67eec7b962bfe82696c2"}
Nov 03 20:39:25.250003 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:25.249943    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq" podStartSLOduration=2.249929711 podStartE2EDuration="2.249929711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:25.245659117 +0000 UTC m=+2295.378787583" watchObservedRunningTime="2026-11-03 20:39:25.249929711 +0000 UTC m=+2295.383058200"
Nov 03 20:39:36.301315 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:39:36.301286    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq"
Nov 03 20:39:49.632282 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:39:49.632239    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:40:19.632836 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:40:19.632799    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:40:49.645913 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:40:49.645795    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:41:19.630579 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:41:19.630541    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:41:27.112664 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:27.112565    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq"]
Nov 03 20:41:32.113070 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:32.113027    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq" podUID="592d56a0-fb52-4715-b639-ca70e92763d6" containerName="payload-processing" containerID="cri-o://a6310627e967f618add0c135c96badf2e50353f91e06aed2ae8c2065e72c7716" gracePeriod=25
Nov 03 20:41:32.418515 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:32.418480    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq"
Nov 03 20:41:32.551842 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:32.551807    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/592d56a0-fb52-4715-b639-ca70e92763d6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/592d56a0-fb52-4715-b639-ca70e92763d6-provider-credentials\") pod \"592d56a0-fb52-4715-b639-ca70e92763d6\" (UID: \"592d56a0-fb52-4715-b639-ca70e92763d6\") "
Nov 03 20:41:32.552069 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:32.552028    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/592d56a0-fb52-4715-b639-ca70e92763d6-kube-api-access-8llpr\" (UniqueName: \"kubernetes.io/projected/592d56a0-fb52-4715-b639-ca70e92763d6-kube-api-access-8llpr\") pod \"592d56a0-fb52-4715-b639-ca70e92763d6\" (UID: \"592d56a0-fb52-4715-b639-ca70e92763d6\") "
Nov 03 20:41:32.552249 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:32.552165    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/592d56a0-fb52-4715-b639-ca70e92763d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592d56a0-fb52-4715-b639-ca70e92763d6-plugins-config-volume\") pod \"592d56a0-fb52-4715-b639-ca70e92763d6\" (UID: \"592d56a0-fb52-4715-b639-ca70e92763d6\") "
Nov 03 20:41:32.552330 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:32.552316    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/592d56a0-fb52-4715-b639-ca70e92763d6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/592d56a0-fb52-4715-b639-ca70e92763d6-routing-overlay-volume\") pod \"592d56a0-fb52-4715-b639-ca70e92763d6\" (UID: \"592d56a0-fb52-4715-b639-ca70e92763d6\") "
Nov 03 20:41:32.553143 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:32.552834    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/592d56a0-fb52-4715-b639-ca70e92763d6-routing-overlay-volume" pod "592d56a0-fb52-4715-b639-ca70e92763d6" (UID: "592d56a0-fb52-4715-b639-ca70e92763d6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:41:32.553143 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:32.552620    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/592d56a0-fb52-4715-b639-ca70e92763d6-plugins-config-volume" pod "592d56a0-fb52-4715-b639-ca70e92763d6" (UID: "592d56a0-fb52-4715-b639-ca70e92763d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:41:32.553143 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:32.553097    4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/592d56a0-fb52-4715-b639-ca70e92763d6-routing-overlay-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:32.553143 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:32.553119    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/592d56a0-fb52-4715-b639-ca70e92763d6-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:32.554589 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:32.554566    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/592d56a0-fb52-4715-b639-ca70e92763d6-provider-credentials" pod "592d56a0-fb52-4715-b639-ca70e92763d6" (UID: "592d56a0-fb52-4715-b639-ca70e92763d6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:41:32.554799 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:32.554716    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/592d56a0-fb52-4715-b639-ca70e92763d6-kube-api-access-8llpr" pod "592d56a0-fb52-4715-b639-ca70e92763d6" (UID: "592d56a0-fb52-4715-b639-ca70e92763d6"). InnerVolumeSpecName "kube-api-access-8llpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:41:32.668530 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:32.668431    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8llpr\" (UniqueName: \"kubernetes.io/projected/592d56a0-fb52-4715-b639-ca70e92763d6-kube-api-access-8llpr\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:32.668799 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:32.668779    4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/592d56a0-fb52-4715-b639-ca70e92763d6-provider-credentials\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:33.220464 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:33.220425    4421 generic.go:356] "Generic (PLEG): container finished" podID="592d56a0-fb52-4715-b639-ca70e92763d6" containerID="a6310627e967f618add0c135c96badf2e50353f91e06aed2ae8c2065e72c7716" exitCode=0
Nov 03 20:41:33.220989 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:33.220479    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq"
Nov 03 20:41:33.220989 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:33.220511    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq" event={"ID":"592d56a0-fb52-4715-b639-ca70e92763d6","Type":"ContainerDied","Data":"a6310627e967f618add0c135c96badf2e50353f91e06aed2ae8c2065e72c7716"}
Nov 03 20:41:33.220989 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:33.220571    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq" event={"ID":"592d56a0-fb52-4715-b639-ca70e92763d6","Type":"ContainerDied","Data":"0145af07853bf495390c6d2a222b007fa491c7e56d4b67eec7b962bfe82696c2"}
Nov 03 20:41:33.220989 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:33.220638    4421 scope.go:122] "RemoveContainer" containerID="a6310627e967f618add0c135c96badf2e50353f91e06aed2ae8c2065e72c7716"
Nov 03 20:41:33.240717 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:33.240435    4421 scope.go:122] "RemoveContainer" containerID="a6310627e967f618add0c135c96badf2e50353f91e06aed2ae8c2065e72c7716"
Nov 03 20:41:33.241041 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:41:33.240987    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6310627e967f618add0c135c96badf2e50353f91e06aed2ae8c2065e72c7716\": container with ID starting with a6310627e967f618add0c135c96badf2e50353f91e06aed2ae8c2065e72c7716 not found: ID does not exist" containerID="a6310627e967f618add0c135c96badf2e50353f91e06aed2ae8c2065e72c7716"
Nov 03 20:41:33.241041 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:33.241021    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6310627e967f618add0c135c96badf2e50353f91e06aed2ae8c2065e72c7716"} err="failed to get container status \"a6310627e967f618add0c135c96badf2e50353f91e06aed2ae8c2065e72c7716\": rpc error: code = NotFound desc = could not find container \"a6310627e967f618add0c135c96badf2e50353f91e06aed2ae8c2065e72c7716\": container with ID starting with a6310627e967f618add0c135c96badf2e50353f91e06aed2ae8c2065e72c7716 not found: ID does not exist"
Nov 03 20:41:33.260348 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:33.260317    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq"]
Nov 03 20:41:33.264550 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:33.264524    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-7d94mmnrq"]
Nov 03 20:41:34.018684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:34.018653    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="592d56a0-fb52-4715-b639-ca70e92763d6" path="/var/lib/kubelet/pods/592d56a0-fb52-4715-b639-ca70e92763d6/volumes"
Nov 03 20:41:43.174288 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:43.174247    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h"]
Nov 03 20:41:43.175154 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:43.175138    4421 cpu_manager.go:414] "containerMap: removing container" podUID="592d56a0-fb52-4715-b639-ca70e92763d6" containerName="payload-processing"
Nov 03 20:41:43.175226 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:43.175156    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="592d56a0-fb52-4715-b639-ca70e92763d6" containerName="payload-processing"
Nov 03 20:41:43.175312 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:43.175303    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="592d56a0-fb52-4715-b639-ca70e92763d6" containerName="payload-processing"
Nov 03 20:41:43.179345 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:43.179327    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h"
Nov 03 20:41:43.187446 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:43.187418    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"payload-processing-external-model-plugins-e2e-extmod-a-37aa95\""
Nov 03 20:41:43.187635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:43.187589    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"openshift-service-ca.crt\""
Nov 03 20:41:43.187707 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:43.187695    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"kube-root-ca.crt\""
Nov 03 20:41:43.187853 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:43.187767    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"routing-overlay\""
Nov 03 20:41:43.187853 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:43.187820    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"agc-cred-233866678b02\""
Nov 03 20:41:43.190979 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:43.190943    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h"]
Nov 03 20:41:43.234953 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:43.234920    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrqrx\" (UniqueName: \"kubernetes.io/projected/0fb06395-28dc-4e30-8344-a3c88b8f0cea-kube-api-access-lrqrx\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h\" (UID: \"0fb06395-28dc-4e30-8344-a3c88b8f0cea\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h"
Nov 03 20:41:43.235118 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:43.234971    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0fb06395-28dc-4e30-8344-a3c88b8f0cea-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h\" (UID: \"0fb06395-28dc-4e30-8344-a3c88b8f0cea\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h"
Nov 03 20:41:43.235162 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:43.235111    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0fb06395-28dc-4e30-8344-a3c88b8f0cea-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h\" (UID: \"0fb06395-28dc-4e30-8344-a3c88b8f0cea\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h"
Nov 03 20:41:43.235262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:43.235236    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fb06395-28dc-4e30-8344-a3c88b8f0cea-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h\" (UID: \"0fb06395-28dc-4e30-8344-a3c88b8f0cea\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h"
Nov 03 20:41:43.336849 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:43.336814    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrqrx\" (UniqueName: \"kubernetes.io/projected/0fb06395-28dc-4e30-8344-a3c88b8f0cea-kube-api-access-lrqrx\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h\" (UID: \"0fb06395-28dc-4e30-8344-a3c88b8f0cea\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h"
Nov 03 20:41:43.337051 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:43.336874    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0fb06395-28dc-4e30-8344-a3c88b8f0cea-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h\" (UID: \"0fb06395-28dc-4e30-8344-a3c88b8f0cea\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h"
Nov 03 20:41:43.337051 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:43.336963    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0fb06395-28dc-4e30-8344-a3c88b8f0cea-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h\" (UID: \"0fb06395-28dc-4e30-8344-a3c88b8f0cea\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h"
Nov 03 20:41:43.337178 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:43.337049    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fb06395-28dc-4e30-8344-a3c88b8f0cea-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h\" (UID: \"0fb06395-28dc-4e30-8344-a3c88b8f0cea\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h"
Nov 03 20:41:43.337807 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:43.337783    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0fb06395-28dc-4e30-8344-a3c88b8f0cea-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h\" (UID: \"0fb06395-28dc-4e30-8344-a3c88b8f0cea\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h"
Nov 03 20:41:43.337891 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:43.337824    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fb06395-28dc-4e30-8344-a3c88b8f0cea-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h\" (UID: \"0fb06395-28dc-4e30-8344-a3c88b8f0cea\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h"
Nov 03 20:41:43.339561 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:43.339537    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0fb06395-28dc-4e30-8344-a3c88b8f0cea-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h\" (UID: \"0fb06395-28dc-4e30-8344-a3c88b8f0cea\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h"
Nov 03 20:41:43.354987 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:43.354958    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrqrx\" (UniqueName: \"kubernetes.io/projected/0fb06395-28dc-4e30-8344-a3c88b8f0cea-kube-api-access-lrqrx\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h\" (UID: \"0fb06395-28dc-4e30-8344-a3c88b8f0cea\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h"
Nov 03 20:41:43.499622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:43.498892    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h"
Nov 03 20:41:44.490377 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:44.490349    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h"]
Nov 03 20:41:44.490705 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:41:44.490373    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb06395_28dc_4e30_8344_a3c88b8f0cea.slice/crio-88b848298e3825445b3c2b0659d5342b4c9aba90d9aec4e5fe8a9307cf0d4bb7 WatchSource:0}: Error finding container 88b848298e3825445b3c2b0659d5342b4c9aba90d9aec4e5fe8a9307cf0d4bb7: Status 404 returned error can't find the container with id 88b848298e3825445b3c2b0659d5342b4c9aba90d9aec4e5fe8a9307cf0d4bb7
Nov 03 20:41:45.324737 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:45.324699    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h" event={"ID":"0fb06395-28dc-4e30-8344-a3c88b8f0cea","Type":"ContainerStarted","Data":"1bc62f661af554016fb402f002b78ab4045e236c9f37461f95f4f8a5c0ef6e10"}
Nov 03 20:41:45.324947 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:45.324766    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h" event={"ID":"0fb06395-28dc-4e30-8344-a3c88b8f0cea","Type":"ContainerStarted","Data":"88b848298e3825445b3c2b0659d5342b4c9aba90d9aec4e5fe8a9307cf0d4bb7"}
Nov 03 20:41:45.324947 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:45.324797    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h"
Nov 03 20:41:45.347548 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:45.347502    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h" podStartSLOduration=2.347487861 podStartE2EDuration="2.347487861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:45.343746899 +0000 UTC m=+2435.476875361" watchObservedRunningTime="2026-11-03 20:41:45.347487861 +0000 UTC m=+2435.480616333"
Nov 03 20:41:49.636301 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:41:49.636265    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:41:56.385620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:41:56.379833    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h"
Nov 03 20:42:00.970646 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:00.962454    4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:42:01.095650 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:01.084757    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:42:01.140624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:01.139709    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:42:01.385786 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:01.380664    4421 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:42:09.201459 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:09.201418    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h"]
Nov 03 20:42:14.202358 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:14.202280    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h" podUID="0fb06395-28dc-4e30-8344-a3c88b8f0cea" containerName="payload-processing" containerID="cri-o://1bc62f661af554016fb402f002b78ab4045e236c9f37461f95f4f8a5c0ef6e10" gracePeriod=25
Nov 03 20:42:14.504143 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:14.504119    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h"
Nov 03 20:42:14.544998 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:14.544972    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fb06395-28dc-4e30-8344-a3c88b8f0cea-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0fb06395-28dc-4e30-8344-a3c88b8f0cea-provider-credentials\") pod \"0fb06395-28dc-4e30-8344-a3c88b8f0cea\" (UID: \"0fb06395-28dc-4e30-8344-a3c88b8f0cea\") "
Nov 03 20:42:14.545303 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:14.545280    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fb06395-28dc-4e30-8344-a3c88b8f0cea-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0fb06395-28dc-4e30-8344-a3c88b8f0cea-routing-overlay-volume\") pod \"0fb06395-28dc-4e30-8344-a3c88b8f0cea\" (UID: \"0fb06395-28dc-4e30-8344-a3c88b8f0cea\") "
Nov 03 20:42:14.545427 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:14.545412    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fb06395-28dc-4e30-8344-a3c88b8f0cea-kube-api-access-lrqrx\" (UniqueName: \"kubernetes.io/projected/0fb06395-28dc-4e30-8344-a3c88b8f0cea-kube-api-access-lrqrx\") pod \"0fb06395-28dc-4e30-8344-a3c88b8f0cea\" (UID: \"0fb06395-28dc-4e30-8344-a3c88b8f0cea\") "
Nov 03 20:42:14.545497 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:14.545439    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fb06395-28dc-4e30-8344-a3c88b8f0cea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fb06395-28dc-4e30-8344-a3c88b8f0cea-plugins-config-volume\") pod \"0fb06395-28dc-4e30-8344-a3c88b8f0cea\" (UID: \"0fb06395-28dc-4e30-8344-a3c88b8f0cea\") "
Nov 03 20:42:14.545702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:14.545672    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fb06395-28dc-4e30-8344-a3c88b8f0cea-routing-overlay-volume" pod "0fb06395-28dc-4e30-8344-a3c88b8f0cea" (UID: "0fb06395-28dc-4e30-8344-a3c88b8f0cea"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:42:14.546124 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:14.546079    4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0fb06395-28dc-4e30-8344-a3c88b8f0cea-routing-overlay-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:42:14.546363 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:14.546337    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fb06395-28dc-4e30-8344-a3c88b8f0cea-plugins-config-volume" pod "0fb06395-28dc-4e30-8344-a3c88b8f0cea" (UID: "0fb06395-28dc-4e30-8344-a3c88b8f0cea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:42:14.547896 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:14.547867    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fb06395-28dc-4e30-8344-a3c88b8f0cea-provider-credentials" pod "0fb06395-28dc-4e30-8344-a3c88b8f0cea" (UID: "0fb06395-28dc-4e30-8344-a3c88b8f0cea"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:42:14.548014 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:14.547939    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fb06395-28dc-4e30-8344-a3c88b8f0cea-kube-api-access-lrqrx" pod "0fb06395-28dc-4e30-8344-a3c88b8f0cea" (UID: "0fb06395-28dc-4e30-8344-a3c88b8f0cea"). InnerVolumeSpecName "kube-api-access-lrqrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:42:14.647334 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:14.647302    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrqrx\" (UniqueName: \"kubernetes.io/projected/0fb06395-28dc-4e30-8344-a3c88b8f0cea-kube-api-access-lrqrx\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:42:14.647334 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:14.647328    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fb06395-28dc-4e30-8344-a3c88b8f0cea-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:42:14.647334 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:14.647337    4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0fb06395-28dc-4e30-8344-a3c88b8f0cea-provider-credentials\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:42:14.827037 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:14.826953    4421 generic.go:356] "Generic (PLEG): container finished" podID="0fb06395-28dc-4e30-8344-a3c88b8f0cea" containerID="1bc62f661af554016fb402f002b78ab4045e236c9f37461f95f4f8a5c0ef6e10" exitCode=0
Nov 03 20:42:14.827037 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:14.826997    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h"
Nov 03 20:42:14.827252 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:14.827035    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h" event={"ID":"0fb06395-28dc-4e30-8344-a3c88b8f0cea","Type":"ContainerDied","Data":"1bc62f661af554016fb402f002b78ab4045e236c9f37461f95f4f8a5c0ef6e10"}
Nov 03 20:42:14.827252 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:14.827095    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h" event={"ID":"0fb06395-28dc-4e30-8344-a3c88b8f0cea","Type":"ContainerDied","Data":"88b848298e3825445b3c2b0659d5342b4c9aba90d9aec4e5fe8a9307cf0d4bb7"}
Nov 03 20:42:14.827252 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:14.827123    4421 scope.go:122] "RemoveContainer" containerID="1bc62f661af554016fb402f002b78ab4045e236c9f37461f95f4f8a5c0ef6e10"
Nov 03 20:42:14.839625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:14.839586    4421 scope.go:122] "RemoveContainer" containerID="1bc62f661af554016fb402f002b78ab4045e236c9f37461f95f4f8a5c0ef6e10"
Nov 03 20:42:14.839925 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:42:14.839902    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bc62f661af554016fb402f002b78ab4045e236c9f37461f95f4f8a5c0ef6e10\": container with ID starting with 1bc62f661af554016fb402f002b78ab4045e236c9f37461f95f4f8a5c0ef6e10 not found: ID does not exist" containerID="1bc62f661af554016fb402f002b78ab4045e236c9f37461f95f4f8a5c0ef6e10"
Nov 03 20:42:14.840015 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:14.839935    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc62f661af554016fb402f002b78ab4045e236c9f37461f95f4f8a5c0ef6e10"} err="failed to get container status \"1bc62f661af554016fb402f002b78ab4045e236c9f37461f95f4f8a5c0ef6e10\": rpc error: code = NotFound desc = could not find container \"1bc62f661af554016fb402f002b78ab4045e236c9f37461f95f4f8a5c0ef6e10\": container with ID starting with 1bc62f661af554016fb402f002b78ab4045e236c9f37461f95f4f8a5c0ef6e10 not found: ID does not exist"
Nov 03 20:42:14.864998 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:14.864955    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h"]
Nov 03 20:42:14.865194 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:14.865028    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fchrt9h"]
Nov 03 20:42:16.018660 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:42:16.018626    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fb06395-28dc-4e30-8344-a3c88b8f0cea" path="/var/lib/kubelet/pods/0fb06395-28dc-4e30-8344-a3c88b8f0cea/volumes"
Nov 03 20:42:19.633929 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:42:19.633526    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:42:49.636041 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:42:49.636000    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:43:12.721658 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:12.721630    4421 scope.go:122] "RemoveContainer" containerID="9ae79e8de8bd63521d257d8ef0968ad1a5cf9e6b42a7f860a8158e961d3ff65f"
Nov 03 20:43:19.641919 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:43:19.641876    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:43:20.461721 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:20.461687    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk"]
Nov 03 20:43:20.462585 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:20.462563    4421 cpu_manager.go:414] "containerMap: removing container" podUID="0fb06395-28dc-4e30-8344-a3c88b8f0cea" containerName="payload-processing"
Nov 03 20:43:20.462585 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:20.462581    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fb06395-28dc-4e30-8344-a3c88b8f0cea" containerName="payload-processing"
Nov 03 20:43:20.462826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:20.462748    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fb06395-28dc-4e30-8344-a3c88b8f0cea" containerName="payload-processing"
Nov 03 20:43:20.468029 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:20.468008    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk"
Nov 03 20:43:20.477177 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:20.477151    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"openshift-service-ca.crt\""
Nov 03 20:43:20.477428 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:20.477255    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"kube-root-ca.crt\""
Nov 03 20:43:20.479537 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:20.478231    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-05473b\""
Nov 03 20:43:20.480584 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:20.480552    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"routing-overlay\""
Nov 03 20:43:20.480584 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:20.480553    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"agc-cred-df3b03b2d83a\""
Nov 03 20:43:20.480807 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:20.480730    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"payload-processing-external-model-e2e--f590b3a5-dockercfg-znqbs\""
Nov 03 20:43:20.482335 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:20.482307    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk"]
Nov 03 20:43:20.625710 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:20.620925    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqr88\" (UniqueName: \"kubernetes.io/projected/00c2249e-6919-4419-81e1-afcccbcaae33-kube-api-access-dqr88\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk\" (UID: \"00c2249e-6919-4419-81e1-afcccbcaae33\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk"
Nov 03 20:43:20.625710 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:20.621101    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/00c2249e-6919-4419-81e1-afcccbcaae33-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk\" (UID: \"00c2249e-6919-4419-81e1-afcccbcaae33\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk"
Nov 03 20:43:20.625710 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:20.621253    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c2249e-6919-4419-81e1-afcccbcaae33-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk\" (UID: \"00c2249e-6919-4419-81e1-afcccbcaae33\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk"
Nov 03 20:43:20.625710 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:20.621296    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00c2249e-6919-4419-81e1-afcccbcaae33-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk\" (UID: \"00c2249e-6919-4419-81e1-afcccbcaae33\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk"
Nov 03 20:43:20.751723 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:20.751627    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c2249e-6919-4419-81e1-afcccbcaae33-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk\" (UID: \"00c2249e-6919-4419-81e1-afcccbcaae33\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk"
Nov 03 20:43:20.771465 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:20.771433    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00c2249e-6919-4419-81e1-afcccbcaae33-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk\" (UID: \"00c2249e-6919-4419-81e1-afcccbcaae33\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk"
Nov 03 20:43:20.771949 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:20.771922    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqr88\" (UniqueName: \"kubernetes.io/projected/00c2249e-6919-4419-81e1-afcccbcaae33-kube-api-access-dqr88\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk\" (UID: \"00c2249e-6919-4419-81e1-afcccbcaae33\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk"
Nov 03 20:43:20.772304 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:20.772275    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/00c2249e-6919-4419-81e1-afcccbcaae33-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk\" (UID: \"00c2249e-6919-4419-81e1-afcccbcaae33\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk"
Nov 03 20:43:20.773806 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:20.773781    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00c2249e-6919-4419-81e1-afcccbcaae33-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk\" (UID: \"00c2249e-6919-4419-81e1-afcccbcaae33\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk"
Nov 03 20:43:20.773948 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:20.752947    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c2249e-6919-4419-81e1-afcccbcaae33-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk\" (UID: \"00c2249e-6919-4419-81e1-afcccbcaae33\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk"
Nov 03 20:43:20.797931 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:20.783912    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/00c2249e-6919-4419-81e1-afcccbcaae33-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk\" (UID: \"00c2249e-6919-4419-81e1-afcccbcaae33\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk"
Nov 03 20:43:20.835117 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:20.835060    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqr88\" (UniqueName: \"kubernetes.io/projected/00c2249e-6919-4419-81e1-afcccbcaae33-kube-api-access-dqr88\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk\" (UID: \"00c2249e-6919-4419-81e1-afcccbcaae33\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk"
Nov 03 20:43:21.114650 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:21.114527    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk"
Nov 03 20:43:21.413166 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:21.413128    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk"]
Nov 03 20:43:21.413386 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:43:21.413361    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c2249e_6919_4419_81e1_afcccbcaae33.slice/crio-ed2d26047d31f3c7648fc4c6354f47614837993c08eed85031a874085e363641 WatchSource:0}: Error finding container ed2d26047d31f3c7648fc4c6354f47614837993c08eed85031a874085e363641: Status 404 returned error can't find the container with id ed2d26047d31f3c7648fc4c6354f47614837993c08eed85031a874085e363641
Nov 03 20:43:21.788674 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:21.788637    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk" event={"ID":"00c2249e-6919-4419-81e1-afcccbcaae33","Type":"ContainerStarted","Data":"f52005b4bd7a38219b2d07f258154ad077f360b28c1cd394cbc42daab2748c66"}
Nov 03 20:43:21.789062 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:21.788702    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk" event={"ID":"00c2249e-6919-4419-81e1-afcccbcaae33","Type":"ContainerStarted","Data":"ed2d26047d31f3c7648fc4c6354f47614837993c08eed85031a874085e363641"}
Nov 03 20:43:21.789062 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:21.788787    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk"
Nov 03 20:43:21.810028 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:21.809981    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk" podStartSLOduration=1.809967723 podStartE2EDuration="1.809967723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:21.807836811 +0000 UTC m=+2531.940965273" watchObservedRunningTime="2026-11-03 20:43:21.809967723 +0000 UTC m=+2531.943096185"
Nov 03 20:43:32.800351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:32.800321    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk"
Nov 03 20:43:49.629888 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:43:49.629805    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:43:57.566627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:57.546631    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-6bhmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:43:57.566627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:57.546713    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr" podUID="9d6e2ee7-7d48-4879-b6fa-f303b5d24697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:43:57.594329 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:57.594293    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf"]
Nov 03 20:43:57.739453 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:57.739418    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf"]
Nov 03 20:43:57.740044 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:57.740020    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf"
Nov 03 20:43:57.979696 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:57.979666    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8c6934-dockercfg-f6wtr\""
Nov 03 20:43:58.003849 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:57.981741    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8c6934\""
Nov 03 20:43:58.003849 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:57.984722    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46ec9f9f-d32d-486e-8c5a-6bc074038ca7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf\" (UID: \"46ec9f9f-d32d-486e-8c5a-6bc074038ca7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf"
Nov 03 20:43:58.003849 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:57.984946    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w42f\" (UniqueName: \"kubernetes.io/projected/46ec9f9f-d32d-486e-8c5a-6bc074038ca7-kube-api-access-7w42f\") pod \"payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf\" (UID: \"46ec9f9f-d32d-486e-8c5a-6bc074038ca7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf"
Nov 03 20:43:58.090467 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:58.090376    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46ec9f9f-d32d-486e-8c5a-6bc074038ca7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf\" (UID: \"46ec9f9f-d32d-486e-8c5a-6bc074038ca7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf"
Nov 03 20:43:58.090665 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:58.090559    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w42f\" (UniqueName: \"kubernetes.io/projected/46ec9f9f-d32d-486e-8c5a-6bc074038ca7-kube-api-access-7w42f\") pod \"payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf\" (UID: \"46ec9f9f-d32d-486e-8c5a-6bc074038ca7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf"
Nov 03 20:43:58.095405 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:58.095194    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46ec9f9f-d32d-486e-8c5a-6bc074038ca7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf\" (UID: \"46ec9f9f-d32d-486e-8c5a-6bc074038ca7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf"
Nov 03 20:43:58.105484 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:58.105456    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w42f\" (UniqueName: \"kubernetes.io/projected/46ec9f9f-d32d-486e-8c5a-6bc074038ca7-kube-api-access-7w42f\") pod \"payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf\" (UID: \"46ec9f9f-d32d-486e-8c5a-6bc074038ca7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf"
Nov 03 20:43:58.312628 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:58.312576    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf"
Nov 03 20:43:58.615561 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:58.615532    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf"]
Nov 03 20:43:59.047459 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:59.047417    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf" event={"ID":"46ec9f9f-d32d-486e-8c5a-6bc074038ca7","Type":"ContainerStarted","Data":"7e2ffbc6ad030625ddd258a518df01a41ccb8e35ef6e787db4497b91849aa616"}
Nov 03 20:43:59.047685 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:59.047511    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf" event={"ID":"46ec9f9f-d32d-486e-8c5a-6bc074038ca7","Type":"ContainerStarted","Data":"8c8a46230ec9d6fabd837f3e3a83ca9cf9231617b10c4046b25f195bfcb67fb1"}
Nov 03 20:43:59.047685 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:59.047542    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf"
Nov 03 20:43:59.076068 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:43:59.076004    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf" podStartSLOduration=3.075985134 podStartE2EDuration="3.075985134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:59.067572468 +0000 UTC m=+2569.200700932" watchObservedRunningTime="2026-11-03 20:43:59.075985134 +0000 UTC m=+2569.209113597"
Nov 03 20:44:02.034620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:02.033920    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45"]
Nov 03 20:44:02.072621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:02.066450    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45"]
Nov 03 20:44:02.072621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:02.066578    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45"
Nov 03 20:44:02.072886 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:02.072834    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3d4fe2-serving-cert\""
Nov 03 20:44:02.212658 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:02.208976    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3b9318c-440f-41d3-8956-837e2ca7ae62-maas-api-tls\") pod \"maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45\" (UID: \"d3b9318c-440f-41d3-8956-837e2ca7ae62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45"
Nov 03 20:44:02.212658 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:02.209063    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3b9318c-440f-41d3-8956-837e2ca7ae62-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45\" (UID: \"d3b9318c-440f-41d3-8956-837e2ca7ae62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45"
Nov 03 20:44:02.212658 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:02.209251    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwfrv\" (UniqueName: \"kubernetes.io/projected/d3b9318c-440f-41d3-8956-837e2ca7ae62-kube-api-access-gwfrv\") pod \"maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45\" (UID: \"d3b9318c-440f-41d3-8956-837e2ca7ae62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45"
Nov 03 20:44:02.212658 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:02.209338    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3b9318c-440f-41d3-8956-837e2ca7ae62-metrics-certs\") pod \"maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45\" (UID: \"d3b9318c-440f-41d3-8956-837e2ca7ae62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45"
Nov 03 20:44:02.311138 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:02.311052    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3b9318c-440f-41d3-8956-837e2ca7ae62-maas-api-tls\") pod \"maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45\" (UID: \"d3b9318c-440f-41d3-8956-837e2ca7ae62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45"
Nov 03 20:44:02.311138 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:02.311126    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3b9318c-440f-41d3-8956-837e2ca7ae62-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45\" (UID: \"d3b9318c-440f-41d3-8956-837e2ca7ae62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45"
Nov 03 20:44:02.311380 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:02.311310    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwfrv\" (UniqueName: \"kubernetes.io/projected/d3b9318c-440f-41d3-8956-837e2ca7ae62-kube-api-access-gwfrv\") pod \"maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45\" (UID: \"d3b9318c-440f-41d3-8956-837e2ca7ae62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45"
Nov 03 20:44:02.311449 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:02.311390    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3b9318c-440f-41d3-8956-837e2ca7ae62-metrics-certs\") pod \"maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45\" (UID: \"d3b9318c-440f-41d3-8956-837e2ca7ae62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45"
Nov 03 20:44:02.311934 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:02.311909    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3b9318c-440f-41d3-8956-837e2ca7ae62-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45\" (UID: \"d3b9318c-440f-41d3-8956-837e2ca7ae62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45"
Nov 03 20:44:02.313822 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:02.313799    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3b9318c-440f-41d3-8956-837e2ca7ae62-maas-api-tls\") pod \"maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45\" (UID: \"d3b9318c-440f-41d3-8956-837e2ca7ae62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45"
Nov 03 20:44:02.314094 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:02.314073    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3b9318c-440f-41d3-8956-837e2ca7ae62-metrics-certs\") pod \"maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45\" (UID: \"d3b9318c-440f-41d3-8956-837e2ca7ae62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45"
Nov 03 20:44:02.323973 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:02.323948    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwfrv\" (UniqueName: \"kubernetes.io/projected/d3b9318c-440f-41d3-8956-837e2ca7ae62-kube-api-access-gwfrv\") pod \"maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45\" (UID: \"d3b9318c-440f-41d3-8956-837e2ca7ae62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45"
Nov 03 20:44:02.404080 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:02.404052    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45"
Nov 03 20:44:02.769883 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:02.764938    4421 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:44:02.770355 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:02.769985    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45"]
Nov 03 20:44:03.243646 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:03.238397    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45" event={"ID":"d3b9318c-440f-41d3-8956-837e2ca7ae62","Type":"ContainerStarted","Data":"4269661eef549e163e1fbd42567419f4efb984d245a4cca993fe82c524bb1f9e"}
Nov 03 20:44:04.249257 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:04.249220    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45" event={"ID":"d3b9318c-440f-41d3-8956-837e2ca7ae62","Type":"ContainerStarted","Data":"53b78541282c1c218146b4b3ab4ee1784b788e2e11b112a77f671b146598a18e"}
Nov 03 20:44:04.249687 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:04.249291    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45"
Nov 03 20:44:04.269960 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:04.269911    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45" podStartSLOduration=1.723578736 podStartE2EDuration="2.269896149s" totalImagesPullingTime="546.317413ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:04.26910541 +0000 UTC m=+2574.402233877" watchObservedRunningTime="2026-11-03 20:44:04.269896149 +0000 UTC m=+2574.403024612"
Nov 03 20:44:10.067345 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:10.067316    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf"
Nov 03 20:44:10.262785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:10.262758    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45"
Nov 03 20:44:15.411683 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:15.409535    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk"]
Nov 03 20:44:15.436656 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:15.435800    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk"]
Nov 03 20:44:15.436656 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:15.435922    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk"
Nov 03 20:44:15.442206 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:15.441009    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-8c6934-serving-cert\""
Nov 03 20:44:15.486977 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:15.486946    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53370acb-4269-4c64-947d-c8f98d5beb9c-metrics-certs\") pod \"maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk\" (UID: \"53370acb-4269-4c64-947d-c8f98d5beb9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk"
Nov 03 20:44:15.487182 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:15.487156    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53370acb-4269-4c64-947d-c8f98d5beb9c-maas-api-tls\") pod \"maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk\" (UID: \"53370acb-4269-4c64-947d-c8f98d5beb9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk"
Nov 03 20:44:15.487313 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:15.487270    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53370acb-4269-4c64-947d-c8f98d5beb9c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk\" (UID: \"53370acb-4269-4c64-947d-c8f98d5beb9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk"
Nov 03 20:44:15.487493 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:15.487471    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmdp9\" (UniqueName: \"kubernetes.io/projected/53370acb-4269-4c64-947d-c8f98d5beb9c-kube-api-access-gmdp9\") pod \"maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk\" (UID: \"53370acb-4269-4c64-947d-c8f98d5beb9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk"
Nov 03 20:44:15.596621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:15.594948    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53370acb-4269-4c64-947d-c8f98d5beb9c-maas-api-tls\") pod \"maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk\" (UID: \"53370acb-4269-4c64-947d-c8f98d5beb9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk"
Nov 03 20:44:15.596621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:15.595043    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53370acb-4269-4c64-947d-c8f98d5beb9c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk\" (UID: \"53370acb-4269-4c64-947d-c8f98d5beb9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk"
Nov 03 20:44:15.596621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:15.595259    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmdp9\" (UniqueName: \"kubernetes.io/projected/53370acb-4269-4c64-947d-c8f98d5beb9c-kube-api-access-gmdp9\") pod \"maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk\" (UID: \"53370acb-4269-4c64-947d-c8f98d5beb9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk"
Nov 03 20:44:15.596621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:15.595428    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53370acb-4269-4c64-947d-c8f98d5beb9c-metrics-certs\") pod \"maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk\" (UID: \"53370acb-4269-4c64-947d-c8f98d5beb9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk"
Nov 03 20:44:15.612618 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:15.606502    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53370acb-4269-4c64-947d-c8f98d5beb9c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk\" (UID: \"53370acb-4269-4c64-947d-c8f98d5beb9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk"
Nov 03 20:44:15.630062 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:15.628657    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53370acb-4269-4c64-947d-c8f98d5beb9c-metrics-certs\") pod \"maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk\" (UID: \"53370acb-4269-4c64-947d-c8f98d5beb9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk"
Nov 03 20:44:15.650422 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:15.648717    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53370acb-4269-4c64-947d-c8f98d5beb9c-maas-api-tls\") pod \"maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk\" (UID: \"53370acb-4269-4c64-947d-c8f98d5beb9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk"
Nov 03 20:44:15.661886 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:15.661742    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmdp9\" (UniqueName: \"kubernetes.io/projected/53370acb-4269-4c64-947d-c8f98d5beb9c-kube-api-access-gmdp9\") pod \"maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk\" (UID: \"53370acb-4269-4c64-947d-c8f98d5beb9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk"
Nov 03 20:44:15.757885 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:15.757849    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk"
Nov 03 20:44:16.155824 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:44:16.153201    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53370acb_4269_4c64_947d_c8f98d5beb9c.slice/crio-e11683f21bc125f2b19fbb7ea11dbdddcc297b25f5a8e4883045fd2a47e7420a WatchSource:0}: Error finding container e11683f21bc125f2b19fbb7ea11dbdddcc297b25f5a8e4883045fd2a47e7420a: Status 404 returned error can't find the container with id e11683f21bc125f2b19fbb7ea11dbdddcc297b25f5a8e4883045fd2a47e7420a
Nov 03 20:44:16.160295 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:16.159228    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk"]
Nov 03 20:44:16.554592 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:16.553188    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk" event={"ID":"53370acb-4269-4c64-947d-c8f98d5beb9c","Type":"ContainerStarted","Data":"e11683f21bc125f2b19fbb7ea11dbdddcc297b25f5a8e4883045fd2a47e7420a"}
Nov 03 20:44:17.562112 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:17.562077    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk" event={"ID":"53370acb-4269-4c64-947d-c8f98d5beb9c","Type":"ContainerStarted","Data":"8470cff6cb70704156ab8ad826db61c0f11f9ea439068086397c8e0ad7bd74f5"}
Nov 03 20:44:17.562856 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:17.562159    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk"
Nov 03 20:44:17.584222 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:17.584176    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk" podStartSLOduration=2.031663701 podStartE2EDuration="2.584163267s" totalImagesPullingTime="552.499566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:17.582676821 +0000 UTC m=+2587.715805286" watchObservedRunningTime="2026-11-03 20:44:17.584163267 +0000 UTC m=+2587.717291746"
Nov 03 20:44:19.634750 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:44:19.634716    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:44:23.576958 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:23.576931    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk"
Nov 03 20:44:33.118461 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:33.118427    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q"]
Nov 03 20:44:33.124775 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:33.124747    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q"
Nov 03 20:44:33.131247 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:33.131218    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q"]
Nov 03 20:44:33.131637 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:33.131574    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c94fdf-dockercfg-9xzvn\""
Nov 03 20:44:33.131741 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:33.131647    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c94fdf\""
Nov 03 20:44:33.187292 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:33.187261    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6kwg\" (UniqueName: \"kubernetes.io/projected/4582fd72-a3a0-4d41-a3a4-29cf500022f4-kube-api-access-f6kwg\") pod \"payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q\" (UID: \"4582fd72-a3a0-4d41-a3a4-29cf500022f4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q"
Nov 03 20:44:33.187633 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:33.187571    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4582fd72-a3a0-4d41-a3a4-29cf500022f4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q\" (UID: \"4582fd72-a3a0-4d41-a3a4-29cf500022f4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q"
Nov 03 20:44:33.289324 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:33.289285    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4582fd72-a3a0-4d41-a3a4-29cf500022f4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q\" (UID: \"4582fd72-a3a0-4d41-a3a4-29cf500022f4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q"
Nov 03 20:44:33.289538 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:33.289519    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6kwg\" (UniqueName: \"kubernetes.io/projected/4582fd72-a3a0-4d41-a3a4-29cf500022f4-kube-api-access-f6kwg\") pod \"payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q\" (UID: \"4582fd72-a3a0-4d41-a3a4-29cf500022f4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q"
Nov 03 20:44:33.290272 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:33.290242    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4582fd72-a3a0-4d41-a3a4-29cf500022f4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q\" (UID: \"4582fd72-a3a0-4d41-a3a4-29cf500022f4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q"
Nov 03 20:44:33.301950 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:33.301925    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6kwg\" (UniqueName: \"kubernetes.io/projected/4582fd72-a3a0-4d41-a3a4-29cf500022f4-kube-api-access-f6kwg\") pod \"payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q\" (UID: \"4582fd72-a3a0-4d41-a3a4-29cf500022f4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q"
Nov 03 20:44:33.456621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:33.448678    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q"
Nov 03 20:44:33.921572 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:33.921544    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q"]
Nov 03 20:44:33.922218 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:44:33.922188    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4582fd72_a3a0_4d41_a3a4_29cf500022f4.slice/crio-652f03a51734675cb96452e663fe2943132887bcce91372f22ab8857a3335128 WatchSource:0}: Error finding container 652f03a51734675cb96452e663fe2943132887bcce91372f22ab8857a3335128: Status 404 returned error can't find the container with id 652f03a51734675cb96452e663fe2943132887bcce91372f22ab8857a3335128
Nov 03 20:44:34.879620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:34.869076    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q" event={"ID":"4582fd72-a3a0-4d41-a3a4-29cf500022f4","Type":"ContainerStarted","Data":"e0172df076c1736fe2d1ec4d682fa9a9e92ed94aefea8513f9afabba321853d5"}
Nov 03 20:44:34.879620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:34.869184    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q" event={"ID":"4582fd72-a3a0-4d41-a3a4-29cf500022f4","Type":"ContainerStarted","Data":"652f03a51734675cb96452e663fe2943132887bcce91372f22ab8857a3335128"}
Nov 03 20:44:34.879620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:34.870444    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q"
Nov 03 20:44:34.895629 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:34.892113    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q" podStartSLOduration=1.892093888 podStartE2EDuration="1.892093888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:34.890570174 +0000 UTC m=+2605.023698644" watchObservedRunningTime="2026-11-03 20:44:34.892093888 +0000 UTC m=+2605.025222352"
Nov 03 20:44:46.775759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:46.775718    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm"]
Nov 03 20:44:46.780851 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:46.780830    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm"
Nov 03 20:44:46.788105 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:46.788080    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f5afd7\""
Nov 03 20:44:46.788255 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:46.788167    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f5afd7-dockercfg-48rjm\""
Nov 03 20:44:46.789870 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:46.789845    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm"]
Nov 03 20:44:46.817643 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:46.817613    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq"]
Nov 03 20:44:46.824662 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:46.824627    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq"
Nov 03 20:44:46.827534 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:46.826490    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq"]
Nov 03 20:44:46.985621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:46.984897    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q"
Nov 03 20:44:47.188238 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:47.186437    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dec8202-6194-4d04-a73a-f9b037827f5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm\" (UID: \"2dec8202-6194-4d04-a73a-f9b037827f5e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm"
Nov 03 20:44:47.188238 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:47.186492    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g56ld\" (UniqueName: \"kubernetes.io/projected/2dec8202-6194-4d04-a73a-f9b037827f5e-kube-api-access-g56ld\") pod \"payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm\" (UID: \"2dec8202-6194-4d04-a73a-f9b037827f5e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm"
Nov 03 20:44:47.188238 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:47.186861    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmmjh\" (UniqueName: \"kubernetes.io/projected/11490efe-bbaa-4384-a3cf-bfccd7b6dca3-kube-api-access-bmmjh\") pod \"payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq\" (UID: \"11490efe-bbaa-4384-a3cf-bfccd7b6dca3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq"
Nov 03 20:44:47.188238 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:47.187157    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11490efe-bbaa-4384-a3cf-bfccd7b6dca3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq\" (UID: \"11490efe-bbaa-4384-a3cf-bfccd7b6dca3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq"
Nov 03 20:44:47.289355 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:47.289321    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dec8202-6194-4d04-a73a-f9b037827f5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm\" (UID: \"2dec8202-6194-4d04-a73a-f9b037827f5e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm"
Nov 03 20:44:47.289554 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:47.289399    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g56ld\" (UniqueName: \"kubernetes.io/projected/2dec8202-6194-4d04-a73a-f9b037827f5e-kube-api-access-g56ld\") pod \"payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm\" (UID: \"2dec8202-6194-4d04-a73a-f9b037827f5e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm"
Nov 03 20:44:47.289678 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:47.289653    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmmjh\" (UniqueName: \"kubernetes.io/projected/11490efe-bbaa-4384-a3cf-bfccd7b6dca3-kube-api-access-bmmjh\") pod \"payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq\" (UID: \"11490efe-bbaa-4384-a3cf-bfccd7b6dca3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq"
Nov 03 20:44:47.290019 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:47.289999    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11490efe-bbaa-4384-a3cf-bfccd7b6dca3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq\" (UID: \"11490efe-bbaa-4384-a3cf-bfccd7b6dca3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq"
Nov 03 20:44:47.290726 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:47.290704    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dec8202-6194-4d04-a73a-f9b037827f5e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm\" (UID: \"2dec8202-6194-4d04-a73a-f9b037827f5e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm"
Nov 03 20:44:47.290831 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:47.290748    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11490efe-bbaa-4384-a3cf-bfccd7b6dca3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq\" (UID: \"11490efe-bbaa-4384-a3cf-bfccd7b6dca3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq"
Nov 03 20:44:47.302743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:47.302723    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmmjh\" (UniqueName: \"kubernetes.io/projected/11490efe-bbaa-4384-a3cf-bfccd7b6dca3-kube-api-access-bmmjh\") pod \"payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq\" (UID: \"11490efe-bbaa-4384-a3cf-bfccd7b6dca3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq"
Nov 03 20:44:47.302916 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:47.302898    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g56ld\" (UniqueName: \"kubernetes.io/projected/2dec8202-6194-4d04-a73a-f9b037827f5e-kube-api-access-g56ld\") pod \"payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm\" (UID: \"2dec8202-6194-4d04-a73a-f9b037827f5e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm"
Nov 03 20:44:47.399771 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:47.399743    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm"
Nov 03 20:44:47.442494 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:47.442403    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq"
Nov 03 20:44:47.743319 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:47.743288    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm"]
Nov 03 20:44:47.753489 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:44:47.753455    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11490efe_bbaa_4384_a3cf_bfccd7b6dca3.slice/crio-3bb0c988504d574bd1a783ca2f97aaa372a4c8a6cd0beef67ee99318cff8a4e8 WatchSource:0}: Error finding container 3bb0c988504d574bd1a783ca2f97aaa372a4c8a6cd0beef67ee99318cff8a4e8: Status 404 returned error can't find the container with id 3bb0c988504d574bd1a783ca2f97aaa372a4c8a6cd0beef67ee99318cff8a4e8
Nov 03 20:44:47.753627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:47.753567    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq"]
Nov 03 20:44:48.226047 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:48.226000    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq" event={"ID":"11490efe-bbaa-4384-a3cf-bfccd7b6dca3","Type":"ContainerStarted","Data":"ae90f48a96c49b8f20c462dc12ff090ea705e4350bb52c2eb877c56e35072314"}
Nov 03 20:44:48.226546 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:48.226098    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq" event={"ID":"11490efe-bbaa-4384-a3cf-bfccd7b6dca3","Type":"ContainerStarted","Data":"3bb0c988504d574bd1a783ca2f97aaa372a4c8a6cd0beef67ee99318cff8a4e8"}
Nov 03 20:44:48.226546 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:48.226524    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq"
Nov 03 20:44:48.229002 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:48.228969    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm" event={"ID":"2dec8202-6194-4d04-a73a-f9b037827f5e","Type":"ContainerStarted","Data":"12049298fc9471031557dc4a899d44ce496fbb2a9dc7a9ca116150156ca8a055"}
Nov 03 20:44:48.229129 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:48.229022    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm" event={"ID":"2dec8202-6194-4d04-a73a-f9b037827f5e","Type":"ContainerStarted","Data":"531e32ea4d45b38d537182a409ac1c75467d46c330194b6ee7c0b22945347ac3"}
Nov 03 20:44:48.229129 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:48.229052    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm"
Nov 03 20:44:48.254519 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:48.254476    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq" podStartSLOduration=2.254463615 podStartE2EDuration="2.254463615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:48.251334239 +0000 UTC m=+2618.384462712" watchObservedRunningTime="2026-11-03 20:44:48.254463615 +0000 UTC m=+2618.387592107"
Nov 03 20:44:48.276344 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:48.276284    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm" podStartSLOduration=2.27626966 podStartE2EDuration="2.27626966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:48.274392722 +0000 UTC m=+2618.407521195" watchObservedRunningTime="2026-11-03 20:44:48.27626966 +0000 UTC m=+2618.409398122"
Nov 03 20:44:49.630724 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:44:49.630684    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:44:59.241273 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:59.241240    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq"
Nov 03 20:44:59.241909 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:44:59.241324    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm"
Nov 03 20:45:00.263028 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:00.262994    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm"]
Nov 03 20:45:00.272285 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:00.272258    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm"
Nov 03 20:45:00.277852 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:00.277784    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd"]
Nov 03 20:45:00.284332 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:00.284308    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd"
Nov 03 20:45:00.285184 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:00.285166    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm"]
Nov 03 20:45:00.293370 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:00.292991    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd"]
Nov 03 20:45:00.307340 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:00.307176    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\""
Nov 03 20:45:00.321618 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:00.321565    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cvpd\" (UniqueName: \"kubernetes.io/projected/54180a27-a67d-4d04-8a17-2f503fdcef5f-kube-api-access-6cvpd\") pod \"maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm\" (UID: \"54180a27-a67d-4d04-8a17-2f503fdcef5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm"
Nov 03 20:45:00.322655 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:00.322626    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54180a27-a67d-4d04-8a17-2f503fdcef5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm\" (UID: \"54180a27-a67d-4d04-8a17-2f503fdcef5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm"
Nov 03 20:45:00.322779 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:00.322750    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4zx2\" (UniqueName: \"kubernetes.io/projected/ac3ff895-7ff7-4a44-a285-2a2468445304-kube-api-access-g4zx2\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd\" (UID: \"ac3ff895-7ff7-4a44-a285-2a2468445304\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd"
Nov 03 20:45:00.322846 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:00.322802    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3ff895-7ff7-4a44-a285-2a2468445304-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd\" (UID: \"ac3ff895-7ff7-4a44-a285-2a2468445304\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd"
Nov 03 20:45:00.425276 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:00.425242    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cvpd\" (UniqueName: \"kubernetes.io/projected/54180a27-a67d-4d04-8a17-2f503fdcef5f-kube-api-access-6cvpd\") pod \"maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm\" (UID: \"54180a27-a67d-4d04-8a17-2f503fdcef5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm"
Nov 03 20:45:00.425489 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:00.425291    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54180a27-a67d-4d04-8a17-2f503fdcef5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm\" (UID: \"54180a27-a67d-4d04-8a17-2f503fdcef5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm"
Nov 03 20:45:00.425489 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:00.425386    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4zx2\" (UniqueName: \"kubernetes.io/projected/ac3ff895-7ff7-4a44-a285-2a2468445304-kube-api-access-g4zx2\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd\" (UID: \"ac3ff895-7ff7-4a44-a285-2a2468445304\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd"
Nov 03 20:45:00.425489 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:00.425433    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3ff895-7ff7-4a44-a285-2a2468445304-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd\" (UID: \"ac3ff895-7ff7-4a44-a285-2a2468445304\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd"
Nov 03 20:45:00.426312 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:00.426287    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3ff895-7ff7-4a44-a285-2a2468445304-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd\" (UID: \"ac3ff895-7ff7-4a44-a285-2a2468445304\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd"
Nov 03 20:45:00.426450 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:00.426312    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54180a27-a67d-4d04-8a17-2f503fdcef5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm\" (UID: \"54180a27-a67d-4d04-8a17-2f503fdcef5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm"
Nov 03 20:45:00.437550 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:00.437527    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cvpd\" (UniqueName: \"kubernetes.io/projected/54180a27-a67d-4d04-8a17-2f503fdcef5f-kube-api-access-6cvpd\") pod \"maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm\" (UID: \"54180a27-a67d-4d04-8a17-2f503fdcef5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm"
Nov 03 20:45:00.437767 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:00.437652    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4zx2\" (UniqueName: \"kubernetes.io/projected/ac3ff895-7ff7-4a44-a285-2a2468445304-kube-api-access-g4zx2\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd\" (UID: \"ac3ff895-7ff7-4a44-a285-2a2468445304\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd"
Nov 03 20:45:00.610376 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:00.610293    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm"
Nov 03 20:45:00.621105 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:00.621080    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd"
Nov 03 20:45:00.905662 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:45:00.905626    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54180a27_a67d_4d04_8a17_2f503fdcef5f.slice/crio-18839f7b575e196967f3c1e7e041602ecdf4f14d578527958f0a94a5f61d09a3 WatchSource:0}: Error finding container 18839f7b575e196967f3c1e7e041602ecdf4f14d578527958f0a94a5f61d09a3: Status 404 returned error can't find the container with id 18839f7b575e196967f3c1e7e041602ecdf4f14d578527958f0a94a5f61d09a3
Nov 03 20:45:00.906072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:00.905991    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm"]
Nov 03 20:45:00.933206 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:45:00.933168    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3ff895_7ff7_4a44_a285_2a2468445304.slice/crio-7364ae068480bc8ff31dd5eee79a1f891cee08e381022aaf5cbb36a0654e5ef3 WatchSource:0}: Error finding container 7364ae068480bc8ff31dd5eee79a1f891cee08e381022aaf5cbb36a0654e5ef3: Status 404 returned error can't find the container with id 7364ae068480bc8ff31dd5eee79a1f891cee08e381022aaf5cbb36a0654e5ef3
Nov 03 20:45:00.982927 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:00.966286    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd"]
Nov 03 20:45:01.353428 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:01.353386    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm" event={"ID":"54180a27-a67d-4d04-8a17-2f503fdcef5f","Type":"ContainerStarted","Data":"18839f7b575e196967f3c1e7e041602ecdf4f14d578527958f0a94a5f61d09a3"}
Nov 03 20:45:01.355509 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:01.355472    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd" event={"ID":"ac3ff895-7ff7-4a44-a285-2a2468445304","Type":"ContainerStarted","Data":"7364ae068480bc8ff31dd5eee79a1f891cee08e381022aaf5cbb36a0654e5ef3"}
Nov 03 20:45:02.366192 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:02.366103    4421 generic.go:356] "Generic (PLEG): container finished" podID="54180a27-a67d-4d04-8a17-2f503fdcef5f" containerID="0a3968ccf1569e2c251f573fdfe55ff6bad40e349a34b365f853cd2cb125c600" exitCode=60
Nov 03 20:45:02.366638 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:02.366189    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm" event={"ID":"54180a27-a67d-4d04-8a17-2f503fdcef5f","Type":"ContainerDied","Data":"0a3968ccf1569e2c251f573fdfe55ff6bad40e349a34b365f853cd2cb125c600"}
Nov 03 20:45:02.366738 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:02.366645    4421 scope.go:122] "RemoveContainer" containerID="0a3968ccf1569e2c251f573fdfe55ff6bad40e349a34b365f853cd2cb125c600"
Nov 03 20:45:02.368072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:02.368049    4421 generic.go:356] "Generic (PLEG): container finished" podID="ac3ff895-7ff7-4a44-a285-2a2468445304" containerID="8c8d8b5ec61a4012a1375918de0b91bca347435937217dd1f830156f9496668d" exitCode=60
Nov 03 20:45:02.368178 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:02.368112    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd" event={"ID":"ac3ff895-7ff7-4a44-a285-2a2468445304","Type":"ContainerDied","Data":"8c8d8b5ec61a4012a1375918de0b91bca347435937217dd1f830156f9496668d"}
Nov 03 20:45:02.368390 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:02.368376    4421 scope.go:122] "RemoveContainer" containerID="8c8d8b5ec61a4012a1375918de0b91bca347435937217dd1f830156f9496668d"
Nov 03 20:45:03.379565 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:03.379526    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm" event={"ID":"54180a27-a67d-4d04-8a17-2f503fdcef5f","Type":"ContainerStarted","Data":"2b0272fdd4dc2afcba1d55808541cdaa88616c8b47383d1994fb3111b908e37b"}
Nov 03 20:45:03.381760 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:03.381724    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd" event={"ID":"ac3ff895-7ff7-4a44-a285-2a2468445304","Type":"ContainerStarted","Data":"5024edb0c8e0fb8c382398de29529aa09761bc460ee4ea586b79710d726f969c"}
Nov 03 20:45:03.399797 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:03.399745    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm" podStartSLOduration=3.399731273 podStartE2EDuration="3.399731273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:03.397424332 +0000 UTC m=+2633.530552794" watchObservedRunningTime="2026-11-03 20:45:03.399731273 +0000 UTC m=+2633.532859772"
Nov 03 20:45:03.417793 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:03.417737    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd" podStartSLOduration=3.417722924 podStartE2EDuration="3.417722924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:03.41463805 +0000 UTC m=+2633.547766511" watchObservedRunningTime="2026-11-03 20:45:03.417722924 +0000 UTC m=+2633.550851385"
Nov 03 20:45:04.391833 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:04.391797    4421 generic.go:356] "Generic (PLEG): container finished" podID="ac3ff895-7ff7-4a44-a285-2a2468445304" containerID="5024edb0c8e0fb8c382398de29529aa09761bc460ee4ea586b79710d726f969c" exitCode=60
Nov 03 20:45:04.392270 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:04.391876    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd" event={"ID":"ac3ff895-7ff7-4a44-a285-2a2468445304","Type":"ContainerDied","Data":"5024edb0c8e0fb8c382398de29529aa09761bc460ee4ea586b79710d726f969c"}
Nov 03 20:45:04.392270 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:04.391945    4421 scope.go:122] "RemoveContainer" containerID="8c8d8b5ec61a4012a1375918de0b91bca347435937217dd1f830156f9496668d"
Nov 03 20:45:04.392270 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:04.392226    4421 scope.go:122] "RemoveContainer" containerID="5024edb0c8e0fb8c382398de29529aa09761bc460ee4ea586b79710d726f969c"
Nov 03 20:45:04.392727 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:45:04.392700    4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd_odh-ai-gateway-infra(ac3ff895-7ff7-4a44-a285-2a2468445304)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd" podUID="ac3ff895-7ff7-4a44-a285-2a2468445304"
Nov 03 20:45:04.394320 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:04.394294    4421 generic.go:356] "Generic (PLEG): container finished" podID="54180a27-a67d-4d04-8a17-2f503fdcef5f" containerID="2b0272fdd4dc2afcba1d55808541cdaa88616c8b47383d1994fb3111b908e37b" exitCode=60
Nov 03 20:45:04.394416 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:04.394345    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm" event={"ID":"54180a27-a67d-4d04-8a17-2f503fdcef5f","Type":"ContainerDied","Data":"2b0272fdd4dc2afcba1d55808541cdaa88616c8b47383d1994fb3111b908e37b"}
Nov 03 20:45:04.394712 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:04.394695    4421 scope.go:122] "RemoveContainer" containerID="2b0272fdd4dc2afcba1d55808541cdaa88616c8b47383d1994fb3111b908e37b"
Nov 03 20:45:04.395070 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:45:04.395050    4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm_odh-ai-gateway-infra(54180a27-a67d-4d04-8a17-2f503fdcef5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm" podUID="54180a27-a67d-4d04-8a17-2f503fdcef5f"
Nov 03 20:45:04.406558 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:04.406533    4421 scope.go:122] "RemoveContainer" containerID="0a3968ccf1569e2c251f573fdfe55ff6bad40e349a34b365f853cd2cb125c600"
Nov 03 20:45:05.404001 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:05.403975    4421 scope.go:122] "RemoveContainer" containerID="5024edb0c8e0fb8c382398de29529aa09761bc460ee4ea586b79710d726f969c"
Nov 03 20:45:05.404456 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:45:05.404354    4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd_odh-ai-gateway-infra(ac3ff895-7ff7-4a44-a285-2a2468445304)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd" podUID="ac3ff895-7ff7-4a44-a285-2a2468445304"
Nov 03 20:45:05.406314 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:05.406294    4421 scope.go:122] "RemoveContainer" containerID="2b0272fdd4dc2afcba1d55808541cdaa88616c8b47383d1994fb3111b908e37b"
Nov 03 20:45:05.406706 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:45:05.406683    4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm_odh-ai-gateway-infra(54180a27-a67d-4d04-8a17-2f503fdcef5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm" podUID="54180a27-a67d-4d04-8a17-2f503fdcef5f"
Nov 03 20:45:05.512316 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:05.512267    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864"]
Nov 03 20:45:05.549986 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:05.549956    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864"]
Nov 03 20:45:05.550381 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:05.550359    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864"
Nov 03 20:45:05.556235 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:05.556215    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f5afd7-serving-cert\""
Nov 03 20:45:05.694620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:05.688817    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf2df26-0976-4da0-b3fa-2b036625fc6e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864\" (UID: \"0cf2df26-0976-4da0-b3fa-2b036625fc6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864"
Nov 03 20:45:05.695221 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:05.695188    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cf2df26-0976-4da0-b3fa-2b036625fc6e-maas-api-tls\") pod \"maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864\" (UID: \"0cf2df26-0976-4da0-b3fa-2b036625fc6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864"
Nov 03 20:45:05.695440 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:05.695424    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwmzg\" (UniqueName: \"kubernetes.io/projected/0cf2df26-0976-4da0-b3fa-2b036625fc6e-kube-api-access-nwmzg\") pod \"maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864\" (UID: \"0cf2df26-0976-4da0-b3fa-2b036625fc6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864"
Nov 03 20:45:05.705993 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:05.695634    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cf2df26-0976-4da0-b3fa-2b036625fc6e-metrics-certs\") pod \"maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864\" (UID: \"0cf2df26-0976-4da0-b3fa-2b036625fc6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864"
Nov 03 20:45:05.808106 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:05.808074    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf2df26-0976-4da0-b3fa-2b036625fc6e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864\" (UID: \"0cf2df26-0976-4da0-b3fa-2b036625fc6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864"
Nov 03 20:45:05.808391 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:05.808366    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cf2df26-0976-4da0-b3fa-2b036625fc6e-maas-api-tls\") pod \"maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864\" (UID: \"0cf2df26-0976-4da0-b3fa-2b036625fc6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864"
Nov 03 20:45:05.808505 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:05.808460    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwmzg\" (UniqueName: \"kubernetes.io/projected/0cf2df26-0976-4da0-b3fa-2b036625fc6e-kube-api-access-nwmzg\") pod \"maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864\" (UID: \"0cf2df26-0976-4da0-b3fa-2b036625fc6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864"
Nov 03 20:45:05.808640 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:05.808576    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cf2df26-0976-4da0-b3fa-2b036625fc6e-metrics-certs\") pod \"maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864\" (UID: \"0cf2df26-0976-4da0-b3fa-2b036625fc6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864"
Nov 03 20:45:05.808894 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:05.808870    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf2df26-0976-4da0-b3fa-2b036625fc6e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864\" (UID: \"0cf2df26-0976-4da0-b3fa-2b036625fc6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864"
Nov 03 20:45:05.811789 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:05.811765    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cf2df26-0976-4da0-b3fa-2b036625fc6e-maas-api-tls\") pod \"maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864\" (UID: \"0cf2df26-0976-4da0-b3fa-2b036625fc6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864"
Nov 03 20:45:05.811873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:05.811843    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cf2df26-0976-4da0-b3fa-2b036625fc6e-metrics-certs\") pod \"maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864\" (UID: \"0cf2df26-0976-4da0-b3fa-2b036625fc6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864"
Nov 03 20:45:05.821031 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:05.821004    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwmzg\" (UniqueName: \"kubernetes.io/projected/0cf2df26-0976-4da0-b3fa-2b036625fc6e-kube-api-access-nwmzg\") pod \"maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864\" (UID: \"0cf2df26-0976-4da0-b3fa-2b036625fc6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864"
Nov 03 20:45:05.869430 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:05.869400    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864"
Nov 03 20:45:06.159960 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:06.159930    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864"]
Nov 03 20:45:06.160770 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:45:06.160741    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf2df26_0976_4da0_b3fa_2b036625fc6e.slice/crio-809138cd119c62a56cd9f65e52edf48b5796949a4d5c555bd6f441286926abd0 WatchSource:0}: Error finding container 809138cd119c62a56cd9f65e52edf48b5796949a4d5c555bd6f441286926abd0: Status 404 returned error can't find the container with id 809138cd119c62a56cd9f65e52edf48b5796949a4d5c555bd6f441286926abd0
Nov 03 20:45:06.418237 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:06.418142    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864" event={"ID":"0cf2df26-0976-4da0-b3fa-2b036625fc6e","Type":"ContainerStarted","Data":"809138cd119c62a56cd9f65e52edf48b5796949a4d5c555bd6f441286926abd0"}
Nov 03 20:45:07.432909 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:07.432875    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864" event={"ID":"0cf2df26-0976-4da0-b3fa-2b036625fc6e","Type":"ContainerStarted","Data":"bb5d5d9e2a6417eae37d80a4504f5ee5826b5edcc25db5fb7f6c211fdd05ade9"}
Nov 03 20:45:07.433320 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:07.432968    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864"
Nov 03 20:45:07.457559 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:07.457424    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864" podStartSLOduration=1.95772885 podStartE2EDuration="2.457408432s" totalImagesPullingTime="499.679582ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:07.452108476 +0000 UTC m=+2637.585236958" watchObservedRunningTime="2026-11-03 20:45:07.457408432 +0000 UTC m=+2637.590536937"
Nov 03 20:45:13.484564 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:13.484530    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864"
Nov 03 20:45:16.008761 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:16.008732    4421 scope.go:122] "RemoveContainer" containerID="5024edb0c8e0fb8c382398de29529aa09761bc460ee4ea586b79710d726f969c"
Nov 03 20:45:16.009150 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:16.008816    4421 scope.go:122] "RemoveContainer" containerID="2b0272fdd4dc2afcba1d55808541cdaa88616c8b47383d1994fb3111b908e37b"
Nov 03 20:45:16.575926 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:16.575882    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd" event={"ID":"ac3ff895-7ff7-4a44-a285-2a2468445304","Type":"ContainerStarted","Data":"a9c35d6731a5b7567dd3ff006470bb6c07babf4f881449e99733324343330fe6"}
Nov 03 20:45:17.586229 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:17.586199    4421 generic.go:356] "Generic (PLEG): container finished" podID="54180a27-a67d-4d04-8a17-2f503fdcef5f" containerID="e617f5433736181de2806548aa74712194f11092ba8340ff9feb8fac28952f95" exitCode=60
Nov 03 20:45:17.586711 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:17.586268    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm" event={"ID":"54180a27-a67d-4d04-8a17-2f503fdcef5f","Type":"ContainerDied","Data":"e617f5433736181de2806548aa74712194f11092ba8340ff9feb8fac28952f95"}
Nov 03 20:45:17.586711 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:17.586328    4421 scope.go:122] "RemoveContainer" containerID="2b0272fdd4dc2afcba1d55808541cdaa88616c8b47383d1994fb3111b908e37b"
Nov 03 20:45:17.586836 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:17.586806    4421 scope.go:122] "RemoveContainer" containerID="e617f5433736181de2806548aa74712194f11092ba8340ff9feb8fac28952f95"
Nov 03 20:45:17.587281 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:45:17.587248    4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm_odh-ai-gateway-infra(54180a27-a67d-4d04-8a17-2f503fdcef5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm" podUID="54180a27-a67d-4d04-8a17-2f503fdcef5f"
Nov 03 20:45:17.588561 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:17.588533    4421 generic.go:356] "Generic (PLEG): container finished" podID="ac3ff895-7ff7-4a44-a285-2a2468445304" containerID="a9c35d6731a5b7567dd3ff006470bb6c07babf4f881449e99733324343330fe6" exitCode=60
Nov 03 20:45:17.588704 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:17.588639    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd" event={"ID":"ac3ff895-7ff7-4a44-a285-2a2468445304","Type":"ContainerDied","Data":"a9c35d6731a5b7567dd3ff006470bb6c07babf4f881449e99733324343330fe6"}
Nov 03 20:45:17.589089 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:17.589070    4421 scope.go:122] "RemoveContainer" containerID="a9c35d6731a5b7567dd3ff006470bb6c07babf4f881449e99733324343330fe6"
Nov 03 20:45:17.589472 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:45:17.589446    4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd_odh-ai-gateway-infra(ac3ff895-7ff7-4a44-a285-2a2468445304)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd" podUID="ac3ff895-7ff7-4a44-a285-2a2468445304"
Nov 03 20:45:17.600519 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:17.600504    4421 scope.go:122] "RemoveContainer" containerID="5024edb0c8e0fb8c382398de29529aa09761bc460ee4ea586b79710d726f969c"
Nov 03 20:45:17.608880 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:17.608856    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd"]
Nov 03 20:45:18.619517 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:18.619493    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm"]
Nov 03 20:45:18.901448 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:18.896686    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd"
Nov 03 20:45:18.927690 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:18.920540    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm"
Nov 03 20:45:19.075172 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:19.075144    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac3ff895-7ff7-4a44-a285-2a2468445304-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3ff895-7ff7-4a44-a285-2a2468445304-maas-api-service-ca\") pod \"ac3ff895-7ff7-4a44-a285-2a2468445304\" (UID: \"ac3ff895-7ff7-4a44-a285-2a2468445304\") "
Nov 03 20:45:19.075422 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:19.075408    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54180a27-a67d-4d04-8a17-2f503fdcef5f-kube-api-access-6cvpd\" (UniqueName: \"kubernetes.io/projected/54180a27-a67d-4d04-8a17-2f503fdcef5f-kube-api-access-6cvpd\") pod \"54180a27-a67d-4d04-8a17-2f503fdcef5f\" (UID: \"54180a27-a67d-4d04-8a17-2f503fdcef5f\") "
Nov 03 20:45:19.075510 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:19.075482    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac3ff895-7ff7-4a44-a285-2a2468445304-maas-api-service-ca" pod "ac3ff895-7ff7-4a44-a285-2a2468445304" (UID: "ac3ff895-7ff7-4a44-a285-2a2468445304"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:19.075577 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:19.075490    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54180a27-a67d-4d04-8a17-2f503fdcef5f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54180a27-a67d-4d04-8a17-2f503fdcef5f-maas-api-service-ca\") pod \"54180a27-a67d-4d04-8a17-2f503fdcef5f\" (UID: \"54180a27-a67d-4d04-8a17-2f503fdcef5f\") "
Nov 03 20:45:19.075677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:19.075636    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac3ff895-7ff7-4a44-a285-2a2468445304-kube-api-access-g4zx2\" (UniqueName: \"kubernetes.io/projected/ac3ff895-7ff7-4a44-a285-2a2468445304-kube-api-access-g4zx2\") pod \"ac3ff895-7ff7-4a44-a285-2a2468445304\" (UID: \"ac3ff895-7ff7-4a44-a285-2a2468445304\") "
Nov 03 20:45:19.075813 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:19.075794    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54180a27-a67d-4d04-8a17-2f503fdcef5f-maas-api-service-ca" pod "54180a27-a67d-4d04-8a17-2f503fdcef5f" (UID: "54180a27-a67d-4d04-8a17-2f503fdcef5f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:19.076387 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:19.076365    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54180a27-a67d-4d04-8a17-2f503fdcef5f-maas-api-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:19.076387 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:19.076389    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3ff895-7ff7-4a44-a285-2a2468445304-maas-api-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:19.077990 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:19.077963    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54180a27-a67d-4d04-8a17-2f503fdcef5f-kube-api-access-6cvpd" pod "54180a27-a67d-4d04-8a17-2f503fdcef5f" (UID: "54180a27-a67d-4d04-8a17-2f503fdcef5f"). InnerVolumeSpecName "kube-api-access-6cvpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:19.078088 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:19.078006    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac3ff895-7ff7-4a44-a285-2a2468445304-kube-api-access-g4zx2" pod "ac3ff895-7ff7-4a44-a285-2a2468445304" (UID: "ac3ff895-7ff7-4a44-a285-2a2468445304"). InnerVolumeSpecName "kube-api-access-g4zx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:19.178721 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:19.178574    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4zx2\" (UniqueName: \"kubernetes.io/projected/ac3ff895-7ff7-4a44-a285-2a2468445304-kube-api-access-g4zx2\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:19.178873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:19.178716    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cvpd\" (UniqueName: \"kubernetes.io/projected/54180a27-a67d-4d04-8a17-2f503fdcef5f-kube-api-access-6cvpd\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:19.687696 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:19.686980    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm" event={"ID":"54180a27-a67d-4d04-8a17-2f503fdcef5f","Type":"ContainerDied","Data":"18839f7b575e196967f3c1e7e041602ecdf4f14d578527958f0a94a5f61d09a3"}
Nov 03 20:45:19.688249 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:19.688228    4421 scope.go:122] "RemoveContainer" containerID="e617f5433736181de2806548aa74712194f11092ba8340ff9feb8fac28952f95"
Nov 03 20:45:19.690730 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:19.687060    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm"
Nov 03 20:45:19.702023 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:19.699971    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd"
Nov 03 20:45:19.702023 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:19.700099    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd" event={"ID":"ac3ff895-7ff7-4a44-a285-2a2468445304","Type":"ContainerDied","Data":"7364ae068480bc8ff31dd5eee79a1f891cee08e381022aaf5cbb36a0654e5ef3"}
Nov 03 20:45:19.772650 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:45:19.770452    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:45:19.790433 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:19.790404    4421 scope.go:122] "RemoveContainer" containerID="a9c35d6731a5b7567dd3ff006470bb6c07babf4f881449e99733324343330fe6"
Nov 03 20:45:19.868236 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:19.868209    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm"]
Nov 03 20:45:19.868431 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:19.868420    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c94fdf-29895645-gkmpm"]
Nov 03 20:45:19.875672 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:19.875641    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd"]
Nov 03 20:45:19.883003 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:19.882969    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895645-lssfd"]
Nov 03 20:45:20.049768 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:20.049669    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54180a27-a67d-4d04-8a17-2f503fdcef5f" path="/var/lib/kubelet/pods/54180a27-a67d-4d04-8a17-2f503fdcef5f/volumes"
Nov 03 20:45:20.050418 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:20.050393    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac3ff895-7ff7-4a44-a285-2a2468445304" path="/var/lib/kubelet/pods/ac3ff895-7ff7-4a44-a285-2a2468445304/volumes"
Nov 03 20:45:37.648753 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:37.648716    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk"]
Nov 03 20:45:42.649165 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:42.649133    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk" podUID="00c2249e-6919-4419-81e1-afcccbcaae33" containerName="payload-processing" containerID="cri-o://f52005b4bd7a38219b2d07f258154ad077f360b28c1cd394cbc42daab2748c66" gracePeriod=25
Nov 03 20:45:42.798199 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:45:42.798167    4421 logging.go:55] [core] [Channel #10080 SubChannel #10081]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.226:50052", ServerName: "10.134.0.226:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.226:50052: connect: connection refused"
Nov 03 20:45:42.996803 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:42.996773    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk"
Nov 03 20:45:43.171944 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:43.171868    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c2249e-6919-4419-81e1-afcccbcaae33-provider-credentials\" (UniqueName: \"kubernetes.io/projected/00c2249e-6919-4419-81e1-afcccbcaae33-provider-credentials\") pod \"00c2249e-6919-4419-81e1-afcccbcaae33\" (UID: \"00c2249e-6919-4419-81e1-afcccbcaae33\") "
Nov 03 20:45:43.172154 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:43.172007    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c2249e-6919-4419-81e1-afcccbcaae33-kube-api-access-dqr88\" (UniqueName: \"kubernetes.io/projected/00c2249e-6919-4419-81e1-afcccbcaae33-kube-api-access-dqr88\") pod \"00c2249e-6919-4419-81e1-afcccbcaae33\" (UID: \"00c2249e-6919-4419-81e1-afcccbcaae33\") "
Nov 03 20:45:43.172345 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:43.172293    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c2249e-6919-4419-81e1-afcccbcaae33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c2249e-6919-4419-81e1-afcccbcaae33-plugins-config-volume\") pod \"00c2249e-6919-4419-81e1-afcccbcaae33\" (UID: \"00c2249e-6919-4419-81e1-afcccbcaae33\") "
Nov 03 20:45:43.172423 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:43.172342    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c2249e-6919-4419-81e1-afcccbcaae33-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00c2249e-6919-4419-81e1-afcccbcaae33-routing-overlay-volume\") pod \"00c2249e-6919-4419-81e1-afcccbcaae33\" (UID: \"00c2249e-6919-4419-81e1-afcccbcaae33\") "
Nov 03 20:45:43.174289 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:43.174257    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c2249e-6919-4419-81e1-afcccbcaae33-routing-overlay-volume" pod "00c2249e-6919-4419-81e1-afcccbcaae33" (UID: "00c2249e-6919-4419-81e1-afcccbcaae33"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:43.174552 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:43.174525    4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00c2249e-6919-4419-81e1-afcccbcaae33-routing-overlay-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:43.175455 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:43.175431    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c2249e-6919-4419-81e1-afcccbcaae33-plugins-config-volume" pod "00c2249e-6919-4419-81e1-afcccbcaae33" (UID: "00c2249e-6919-4419-81e1-afcccbcaae33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:43.178241 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:43.178214    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c2249e-6919-4419-81e1-afcccbcaae33-provider-credentials" pod "00c2249e-6919-4419-81e1-afcccbcaae33" (UID: "00c2249e-6919-4419-81e1-afcccbcaae33"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:43.195715 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:43.188539    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c2249e-6919-4419-81e1-afcccbcaae33-kube-api-access-dqr88" pod "00c2249e-6919-4419-81e1-afcccbcaae33" (UID: "00c2249e-6919-4419-81e1-afcccbcaae33"). InnerVolumeSpecName "kube-api-access-dqr88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:43.195715 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:43.188768    4421 generic.go:356] "Generic (PLEG): container finished" podID="00c2249e-6919-4419-81e1-afcccbcaae33" containerID="f52005b4bd7a38219b2d07f258154ad077f360b28c1cd394cbc42daab2748c66" exitCode=0
Nov 03 20:45:43.195715 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:43.189049    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk" event={"ID":"00c2249e-6919-4419-81e1-afcccbcaae33","Type":"ContainerDied","Data":"f52005b4bd7a38219b2d07f258154ad077f360b28c1cd394cbc42daab2748c66"}
Nov 03 20:45:43.195715 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:43.189126    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk" event={"ID":"00c2249e-6919-4419-81e1-afcccbcaae33","Type":"ContainerDied","Data":"ed2d26047d31f3c7648fc4c6354f47614837993c08eed85031a874085e363641"}
Nov 03 20:45:43.195715 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:43.189166    4421 scope.go:122] "RemoveContainer" containerID="f52005b4bd7a38219b2d07f258154ad077f360b28c1cd394cbc42daab2748c66"
Nov 03 20:45:43.195715 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:43.189351    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk"
Nov 03 20:45:43.220992 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:43.220973    4421 scope.go:122] "RemoveContainer" containerID="f52005b4bd7a38219b2d07f258154ad077f360b28c1cd394cbc42daab2748c66"
Nov 03 20:45:43.221300 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:45:43.221279    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f52005b4bd7a38219b2d07f258154ad077f360b28c1cd394cbc42daab2748c66\": container with ID starting with f52005b4bd7a38219b2d07f258154ad077f360b28c1cd394cbc42daab2748c66 not found: ID does not exist" containerID="f52005b4bd7a38219b2d07f258154ad077f360b28c1cd394cbc42daab2748c66"
Nov 03 20:45:43.221386 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:43.221310    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52005b4bd7a38219b2d07f258154ad077f360b28c1cd394cbc42daab2748c66"} err="failed to get container status \"f52005b4bd7a38219b2d07f258154ad077f360b28c1cd394cbc42daab2748c66\": rpc error: code = NotFound desc = could not find container \"f52005b4bd7a38219b2d07f258154ad077f360b28c1cd394cbc42daab2748c66\": container with ID starting with f52005b4bd7a38219b2d07f258154ad077f360b28c1cd394cbc42daab2748c66 not found: ID does not exist"
Nov 03 20:45:43.234894 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:43.234870    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk"]
Nov 03 20:45:43.243520 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:43.243490    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk"]
Nov 03 20:45:43.276714 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:43.276680    4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/00c2249e-6919-4419-81e1-afcccbcaae33-provider-credentials\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:43.276714 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:43.276705    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqr88\" (UniqueName: \"kubernetes.io/projected/00c2249e-6919-4419-81e1-afcccbcaae33-kube-api-access-dqr88\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:43.276714 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:43.276716    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c2249e-6919-4419-81e1-afcccbcaae33-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:43.797924 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:43.797882    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-57fdljtdk" podUID="00c2249e-6919-4419-81e1-afcccbcaae33" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.226:50052\" within 1s: context deadline exceeded"
Nov 03 20:45:44.065862 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:44.065780    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00c2249e-6919-4419-81e1-afcccbcaae33" path="/var/lib/kubelet/pods/00c2249e-6919-4419-81e1-afcccbcaae33/volumes"
Nov 03 20:45:49.662180 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:45:49.662142    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:45:58.646710 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.646671    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv"]
Nov 03 20:45:58.648033 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.648010    4421 cpu_manager.go:414] "containerMap: removing container" podUID="ac3ff895-7ff7-4a44-a285-2a2468445304" containerName="cleanup"
Nov 03 20:45:58.648033 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.648032    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac3ff895-7ff7-4a44-a285-2a2468445304" containerName="cleanup"
Nov 03 20:45:58.648262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.648052    4421 cpu_manager.go:414] "containerMap: removing container" podUID="00c2249e-6919-4419-81e1-afcccbcaae33" containerName="payload-processing"
Nov 03 20:45:58.648262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.648060    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c2249e-6919-4419-81e1-afcccbcaae33" containerName="payload-processing"
Nov 03 20:45:58.648262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.648084    4421 cpu_manager.go:414] "containerMap: removing container" podUID="ac3ff895-7ff7-4a44-a285-2a2468445304" containerName="cleanup"
Nov 03 20:45:58.648262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.648091    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac3ff895-7ff7-4a44-a285-2a2468445304" containerName="cleanup"
Nov 03 20:45:58.648262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.648099    4421 cpu_manager.go:414] "containerMap: removing container" podUID="ac3ff895-7ff7-4a44-a285-2a2468445304" containerName="cleanup"
Nov 03 20:45:58.648262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.648106    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac3ff895-7ff7-4a44-a285-2a2468445304" containerName="cleanup"
Nov 03 20:45:58.648262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.648120    4421 cpu_manager.go:414] "containerMap: removing container" podUID="54180a27-a67d-4d04-8a17-2f503fdcef5f" containerName="cleanup"
Nov 03 20:45:58.648262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.648127    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54180a27-a67d-4d04-8a17-2f503fdcef5f" containerName="cleanup"
Nov 03 20:45:58.648262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.648159    4421 cpu_manager.go:414] "containerMap: removing container" podUID="54180a27-a67d-4d04-8a17-2f503fdcef5f" containerName="cleanup"
Nov 03 20:45:58.648262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.648166    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54180a27-a67d-4d04-8a17-2f503fdcef5f" containerName="cleanup"
Nov 03 20:45:58.648262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.648208    4421 cpu_manager.go:414] "containerMap: removing container" podUID="54180a27-a67d-4d04-8a17-2f503fdcef5f" containerName="cleanup"
Nov 03 20:45:58.648262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.648217    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54180a27-a67d-4d04-8a17-2f503fdcef5f" containerName="cleanup"
Nov 03 20:45:58.648659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.648403    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="54180a27-a67d-4d04-8a17-2f503fdcef5f" containerName="cleanup"
Nov 03 20:45:58.648659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.648417    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="54180a27-a67d-4d04-8a17-2f503fdcef5f" containerName="cleanup"
Nov 03 20:45:58.648659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.648427    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="54180a27-a67d-4d04-8a17-2f503fdcef5f" containerName="cleanup"
Nov 03 20:45:58.648659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.648437    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac3ff895-7ff7-4a44-a285-2a2468445304" containerName="cleanup"
Nov 03 20:45:58.648659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.648443    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c2249e-6919-4419-81e1-afcccbcaae33" containerName="payload-processing"
Nov 03 20:45:58.648659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.648452    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac3ff895-7ff7-4a44-a285-2a2468445304" containerName="cleanup"
Nov 03 20:45:58.648659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.648458    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac3ff895-7ff7-4a44-a285-2a2468445304" containerName="cleanup"
Nov 03 20:45:58.654271 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.654246    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv"
Nov 03 20:45:58.659614 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.659566    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"agc-cred-159b624266ae\""
Nov 03 20:45:58.659614 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.659573    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"openshift-service-ca.crt\""
Nov 03 20:45:58.659807 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.659792    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"routing-overlay\""
Nov 03 20:45:58.659961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.659887    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"payload-processing-external-model-e2e--f590b3a5-dockercfg-5269j\""
Nov 03 20:45:58.659961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.659930    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-05473b\""
Nov 03 20:45:58.661087 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.661065    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv"]
Nov 03 20:45:58.777030 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.776997    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-05473b\"/\"kube-root-ca.crt\""
Nov 03 20:45:58.861959 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.861928    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78dabb1d-ff1a-4de3-a082-cea203b39bea-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv\" (UID: \"78dabb1d-ff1a-4de3-a082-cea203b39bea\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv"
Nov 03 20:45:58.862294 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.862271    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/78dabb1d-ff1a-4de3-a082-cea203b39bea-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv\" (UID: \"78dabb1d-ff1a-4de3-a082-cea203b39bea\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv"
Nov 03 20:45:58.862527 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.862509    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/78dabb1d-ff1a-4de3-a082-cea203b39bea-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv\" (UID: \"78dabb1d-ff1a-4de3-a082-cea203b39bea\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv"
Nov 03 20:45:58.862809 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:58.862787    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srtr5\" (UniqueName: \"kubernetes.io/projected/78dabb1d-ff1a-4de3-a082-cea203b39bea-kube-api-access-srtr5\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv\" (UID: \"78dabb1d-ff1a-4de3-a082-cea203b39bea\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv"
Nov 03 20:45:59.049624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:59.041019    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78dabb1d-ff1a-4de3-a082-cea203b39bea-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv\" (UID: \"78dabb1d-ff1a-4de3-a082-cea203b39bea\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv"
Nov 03 20:45:59.049624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:59.041098    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/78dabb1d-ff1a-4de3-a082-cea203b39bea-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv\" (UID: \"78dabb1d-ff1a-4de3-a082-cea203b39bea\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv"
Nov 03 20:45:59.049624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:59.041158    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/78dabb1d-ff1a-4de3-a082-cea203b39bea-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv\" (UID: \"78dabb1d-ff1a-4de3-a082-cea203b39bea\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv"
Nov 03 20:45:59.049624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:59.041245    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srtr5\" (UniqueName: \"kubernetes.io/projected/78dabb1d-ff1a-4de3-a082-cea203b39bea-kube-api-access-srtr5\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv\" (UID: \"78dabb1d-ff1a-4de3-a082-cea203b39bea\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv"
Nov 03 20:45:59.134658 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:59.131784    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78dabb1d-ff1a-4de3-a082-cea203b39bea-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv\" (UID: \"78dabb1d-ff1a-4de3-a082-cea203b39bea\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv"
Nov 03 20:45:59.134658 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:59.132554    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/78dabb1d-ff1a-4de3-a082-cea203b39bea-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv\" (UID: \"78dabb1d-ff1a-4de3-a082-cea203b39bea\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv"
Nov 03 20:45:59.325649 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:59.325547    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/78dabb1d-ff1a-4de3-a082-cea203b39bea-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv\" (UID: \"78dabb1d-ff1a-4de3-a082-cea203b39bea\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv"
Nov 03 20:45:59.412065 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:59.412019    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srtr5\" (UniqueName: \"kubernetes.io/projected/78dabb1d-ff1a-4de3-a082-cea203b39bea-kube-api-access-srtr5\") pod \"payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv\" (UID: \"78dabb1d-ff1a-4de3-a082-cea203b39bea\") " pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv"
Nov 03 20:45:59.572680 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:59.572646    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv"
Nov 03 20:45:59.872308 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:45:59.872276    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv"]
Nov 03 20:45:59.872775 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:45:59.872750    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78dabb1d_ff1a_4de3_a082_cea203b39bea.slice/crio-f680daf117a4a03464f67fbec1b89ccaa53a49eaa228fb92ce5a6ba8b725b484 WatchSource:0}: Error finding container f680daf117a4a03464f67fbec1b89ccaa53a49eaa228fb92ce5a6ba8b725b484: Status 404 returned error can't find the container with id f680daf117a4a03464f67fbec1b89ccaa53a49eaa228fb92ce5a6ba8b725b484
Nov 03 20:46:00.548943 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:00.548907    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv" event={"ID":"78dabb1d-ff1a-4de3-a082-cea203b39bea","Type":"ContainerStarted","Data":"3426b4f9c962d697e7ca0852ec56221a83508f062b02349afdf137586ac860ce"}
Nov 03 20:46:00.549135 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:00.548962    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv" event={"ID":"78dabb1d-ff1a-4de3-a082-cea203b39bea","Type":"ContainerStarted","Data":"f680daf117a4a03464f67fbec1b89ccaa53a49eaa228fb92ce5a6ba8b725b484"}
Nov 03 20:46:00.549135 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:00.548987    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv"
Nov 03 20:46:00.573207 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:00.573159    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv" podStartSLOduration=2.573145183 podStartE2EDuration="2.573145183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:00.568305678 +0000 UTC m=+2690.701434173" watchObservedRunningTime="2026-11-03 20:46:00.573145183 +0000 UTC m=+2690.706273645"
Nov 03 20:46:11.560366 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:11.560335    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv"
Nov 03 20:46:19.756973 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:19.756924    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:46:31.437044 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:31.437006    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"]
Nov 03 20:46:31.437454 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:31.437208    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7" podUID="3cf17e0a-2b24-4633-afb7-44eebab411d3" containerName="istio-proxy" containerID="cri-o://60856f4ccb96c1b6dfbc9a8dea5e05dff094720bc3b796e039dbc3a8e57c13cf" gracePeriod=30
Nov 03 20:46:32.204228 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:32.204190    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"]
Nov 03 20:46:32.204443 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:32.204421    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4" podUID="830b16e0-459d-444c-b7da-c8d6d8576d4f" containerName="istio-proxy" containerID="cri-o://57d92c1a1e5f97709e501e78b53a1cda2b96502f2d7197edce56424ab1b29889" gracePeriod=30
Nov 03 20:46:32.809589 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:32.809553    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"]
Nov 03 20:46:32.810032 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:32.809834    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw" podUID="317e6b68-68f8-4220-a3ef-2f0999a19ad9" containerName="istio-proxy" containerID="cri-o://2b14fce9a07d8e0822f9cb6964c3462892594276505a91b7dc0a9ff28e28e167" gracePeriod=30
Nov 03 20:46:36.744705 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.744675    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:46:36.778169 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.778141    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-token\" (UniqueName: \"kubernetes.io/projected/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-token\") pod \"3cf17e0a-2b24-4633-afb7-44eebab411d3\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") "
Nov 03 20:46:36.778345 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.778205    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-workload-certs\") pod \"3cf17e0a-2b24-4633-afb7-44eebab411d3\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") "
Nov 03 20:46:36.778345 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.778263    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cf17e0a-2b24-4633-afb7-44eebab411d3-kube-api-access-vmqx5\" (UniqueName: \"kubernetes.io/projected/3cf17e0a-2b24-4633-afb7-44eebab411d3-kube-api-access-vmqx5\") pod \"3cf17e0a-2b24-4633-afb7-44eebab411d3\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") "
Nov 03 20:46:36.778345 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.778335    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cf17e0a-2b24-4633-afb7-44eebab411d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf17e0a-2b24-4633-afb7-44eebab411d3-istiod-ca-cert\") pod \"3cf17e0a-2b24-4633-afb7-44eebab411d3\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") "
Nov 03 20:46:36.778544 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.778448    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-podinfo\") pod \"3cf17e0a-2b24-4633-afb7-44eebab411d3\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") "
Nov 03 20:46:36.778544 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.778475    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-workload-socket\") pod \"3cf17e0a-2b24-4633-afb7-44eebab411d3\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") "
Nov 03 20:46:36.778544 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.778524    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-workload-certs" pod "3cf17e0a-2b24-4633-afb7-44eebab411d3" (UID: "3cf17e0a-2b24-4633-afb7-44eebab411d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:36.778747 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.778650    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-envoy\") pod \"3cf17e0a-2b24-4633-afb7-44eebab411d3\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") "
Nov 03 20:46:36.778805 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.778767    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-credential-socket\") pod \"3cf17e0a-2b24-4633-afb7-44eebab411d3\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") "
Nov 03 20:46:36.778873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.778804    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-data\") pod \"3cf17e0a-2b24-4633-afb7-44eebab411d3\" (UID: \"3cf17e0a-2b24-4633-afb7-44eebab411d3\") "
Nov 03 20:46:36.779798 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.778873    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-workload-socket" pod "3cf17e0a-2b24-4633-afb7-44eebab411d3" (UID: "3cf17e0a-2b24-4633-afb7-44eebab411d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:36.779798 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.778997    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cf17e0a-2b24-4633-afb7-44eebab411d3-istiod-ca-cert" pod "3cf17e0a-2b24-4633-afb7-44eebab411d3" (UID: "3cf17e0a-2b24-4633-afb7-44eebab411d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:46:36.779798 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.779316    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-credential-socket" pod "3cf17e0a-2b24-4633-afb7-44eebab411d3" (UID: "3cf17e0a-2b24-4633-afb7-44eebab411d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:36.779798 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.779468    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-data" pod "3cf17e0a-2b24-4633-afb7-44eebab411d3" (UID: "3cf17e0a-2b24-4633-afb7-44eebab411d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:36.780067 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.779844    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:36.780067 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.779861    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:36.780067 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.779876    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:36.780067 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.779885    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:36.780067 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.779894    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf17e0a-2b24-4633-afb7-44eebab411d3-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:36.780901 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.780861    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-token" pod "3cf17e0a-2b24-4633-afb7-44eebab411d3" (UID: "3cf17e0a-2b24-4633-afb7-44eebab411d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:46:36.781203 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.781174    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-podinfo" pod "3cf17e0a-2b24-4633-afb7-44eebab411d3" (UID: "3cf17e0a-2b24-4633-afb7-44eebab411d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:46:36.782359 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.782330    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-envoy" pod "3cf17e0a-2b24-4633-afb7-44eebab411d3" (UID: "3cf17e0a-2b24-4633-afb7-44eebab411d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:36.782679 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.782656    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf17e0a-2b24-4633-afb7-44eebab411d3-kube-api-access-vmqx5" pod "3cf17e0a-2b24-4633-afb7-44eebab411d3" (UID: "3cf17e0a-2b24-4633-afb7-44eebab411d3"). InnerVolumeSpecName "kube-api-access-vmqx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:46:36.890876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.888313    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:36.890876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.888352    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmqx5\" (UniqueName: \"kubernetes.io/projected/3cf17e0a-2b24-4633-afb7-44eebab411d3-kube-api-access-vmqx5\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:36.890876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.888386    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:36.890876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:36.888401    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf17e0a-2b24-4633-afb7-44eebab411d3-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:37.014522 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.014437    4421 generic.go:356] "Generic (PLEG): container finished" podID="3cf17e0a-2b24-4633-afb7-44eebab411d3" containerID="60856f4ccb96c1b6dfbc9a8dea5e05dff094720bc3b796e039dbc3a8e57c13cf" exitCode=0
Nov 03 20:46:37.014744 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.014550    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7" event={"ID":"3cf17e0a-2b24-4633-afb7-44eebab411d3","Type":"ContainerDied","Data":"60856f4ccb96c1b6dfbc9a8dea5e05dff094720bc3b796e039dbc3a8e57c13cf"}
Nov 03 20:46:37.014744 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.014586    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7" event={"ID":"3cf17e0a-2b24-4633-afb7-44eebab411d3","Type":"ContainerDied","Data":"85a14b9558e5084c6c14441456302ebf7a3e4f8bda7bc59b30080ec274ba8e8c"}
Nov 03 20:46:37.014744 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.014641    4421 scope.go:122] "RemoveContainer" containerID="60856f4ccb96c1b6dfbc9a8dea5e05dff094720bc3b796e039dbc3a8e57c13cf"
Nov 03 20:46:37.014978 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.014749    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"
Nov 03 20:46:37.028665 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.028642    4421 scope.go:122] "RemoveContainer" containerID="60856f4ccb96c1b6dfbc9a8dea5e05dff094720bc3b796e039dbc3a8e57c13cf"
Nov 03 20:46:37.028924 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:37.028907    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60856f4ccb96c1b6dfbc9a8dea5e05dff094720bc3b796e039dbc3a8e57c13cf\": container with ID starting with 60856f4ccb96c1b6dfbc9a8dea5e05dff094720bc3b796e039dbc3a8e57c13cf not found: ID does not exist" containerID="60856f4ccb96c1b6dfbc9a8dea5e05dff094720bc3b796e039dbc3a8e57c13cf"
Nov 03 20:46:37.028997 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.028937    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60856f4ccb96c1b6dfbc9a8dea5e05dff094720bc3b796e039dbc3a8e57c13cf"} err="failed to get container status \"60856f4ccb96c1b6dfbc9a8dea5e05dff094720bc3b796e039dbc3a8e57c13cf\": rpc error: code = NotFound desc = could not find container \"60856f4ccb96c1b6dfbc9a8dea5e05dff094720bc3b796e039dbc3a8e57c13cf\": container with ID starting with 60856f4ccb96c1b6dfbc9a8dea5e05dff094720bc3b796e039dbc3a8e57c13cf not found: ID does not exist"
Nov 03 20:46:37.043654 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.043582    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"]
Nov 03 20:46:37.048077 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.048054    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-8vnp7"]
Nov 03 20:46:37.497490 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.497466    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:46:37.602099 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.602007    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-data\") pod \"830b16e0-459d-444c-b7da-c8d6d8576d4f\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") "
Nov 03 20:46:37.602279 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.602156    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-workload-socket\") pod \"830b16e0-459d-444c-b7da-c8d6d8576d4f\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") "
Nov 03 20:46:37.602279 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.602251    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-credential-socket\") pod \"830b16e0-459d-444c-b7da-c8d6d8576d4f\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") "
Nov 03 20:46:37.602405 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.602297    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-workload-certs\") pod \"830b16e0-459d-444c-b7da-c8d6d8576d4f\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") "
Nov 03 20:46:37.602465 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.602407    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-token\" (UniqueName: \"kubernetes.io/projected/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-token\") pod \"830b16e0-459d-444c-b7da-c8d6d8576d4f\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") "
Nov 03 20:46:37.602465 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.602295    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-data" pod "830b16e0-459d-444c-b7da-c8d6d8576d4f" (UID: "830b16e0-459d-444c-b7da-c8d6d8576d4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:37.602465 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.602422    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-workload-socket" pod "830b16e0-459d-444c-b7da-c8d6d8576d4f" (UID: "830b16e0-459d-444c-b7da-c8d6d8576d4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:37.602637 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.602466    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/830b16e0-459d-444c-b7da-c8d6d8576d4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/830b16e0-459d-444c-b7da-c8d6d8576d4f-istiod-ca-cert\") pod \"830b16e0-459d-444c-b7da-c8d6d8576d4f\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") "
Nov 03 20:46:37.602637 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.602505    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-podinfo\") pod \"830b16e0-459d-444c-b7da-c8d6d8576d4f\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") "
Nov 03 20:46:37.602637 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.602545    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-envoy\") pod \"830b16e0-459d-444c-b7da-c8d6d8576d4f\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") "
Nov 03 20:46:37.602637 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.602614    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-credential-socket" pod "830b16e0-459d-444c-b7da-c8d6d8576d4f" (UID: "830b16e0-459d-444c-b7da-c8d6d8576d4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:37.602637 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.602629    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-workload-certs" pod "830b16e0-459d-444c-b7da-c8d6d8576d4f" (UID: "830b16e0-459d-444c-b7da-c8d6d8576d4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:37.602922 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.602715    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/830b16e0-459d-444c-b7da-c8d6d8576d4f-kube-api-access-x8hqw\" (UniqueName: \"kubernetes.io/projected/830b16e0-459d-444c-b7da-c8d6d8576d4f-kube-api-access-x8hqw\") pod \"830b16e0-459d-444c-b7da-c8d6d8576d4f\" (UID: \"830b16e0-459d-444c-b7da-c8d6d8576d4f\") "
Nov 03 20:46:37.602922 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.602831    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/830b16e0-459d-444c-b7da-c8d6d8576d4f-istiod-ca-cert" pod "830b16e0-459d-444c-b7da-c8d6d8576d4f" (UID: "830b16e0-459d-444c-b7da-c8d6d8576d4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:46:37.603523 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.603499    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:37.603724 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.603702    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/830b16e0-459d-444c-b7da-c8d6d8576d4f-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:37.603724 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.603722    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:37.603872 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.603731    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:37.603872 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.603747    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:37.605178 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.605151    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-podinfo" pod "830b16e0-459d-444c-b7da-c8d6d8576d4f" (UID: "830b16e0-459d-444c-b7da-c8d6d8576d4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:46:37.605281 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.605262    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/830b16e0-459d-444c-b7da-c8d6d8576d4f-kube-api-access-x8hqw" pod "830b16e0-459d-444c-b7da-c8d6d8576d4f" (UID: "830b16e0-459d-444c-b7da-c8d6d8576d4f"). InnerVolumeSpecName "kube-api-access-x8hqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:46:37.605466 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.605451    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-token" pod "830b16e0-459d-444c-b7da-c8d6d8576d4f" (UID: "830b16e0-459d-444c-b7da-c8d6d8576d4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:46:37.605510 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.605465    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-envoy" pod "830b16e0-459d-444c-b7da-c8d6d8576d4f" (UID: "830b16e0-459d-444c-b7da-c8d6d8576d4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:37.705336 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.705304    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:37.705336 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.705329    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:37.705336 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.705339    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/830b16e0-459d-444c-b7da-c8d6d8576d4f-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:37.705574 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:37.705348    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8hqw\" (UniqueName: \"kubernetes.io/projected/830b16e0-459d-444c-b7da-c8d6d8576d4f-kube-api-access-x8hqw\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:38.100545 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.099054    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cf17e0a-2b24-4633-afb7-44eebab411d3" path="/var/lib/kubelet/pods/3cf17e0a-2b24-4633-afb7-44eebab411d3/volumes"
Nov 03 20:46:38.147088 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.147051    4421 generic.go:356] "Generic (PLEG): container finished" podID="317e6b68-68f8-4220-a3ef-2f0999a19ad9" containerID="2b14fce9a07d8e0822f9cb6964c3462892594276505a91b7dc0a9ff28e28e167" exitCode=0
Nov 03 20:46:38.147350 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.147175    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw" event={"ID":"317e6b68-68f8-4220-a3ef-2f0999a19ad9","Type":"ContainerDied","Data":"2b14fce9a07d8e0822f9cb6964c3462892594276505a91b7dc0a9ff28e28e167"}
Nov 03 20:46:38.147350 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.147233    4421 scope.go:122] "RemoveContainer" containerID="759d180f655772ea7fe565676cfd10e5575a2ae5a37aa2bb9454a3c57baee927"
Nov 03 20:46:38.153290 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.153254    4421 generic.go:356] "Generic (PLEG): container finished" podID="830b16e0-459d-444c-b7da-c8d6d8576d4f" containerID="57d92c1a1e5f97709e501e78b53a1cda2b96502f2d7197edce56424ab1b29889" exitCode=0
Nov 03 20:46:38.153428 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.153394    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4" event={"ID":"830b16e0-459d-444c-b7da-c8d6d8576d4f","Type":"ContainerDied","Data":"57d92c1a1e5f97709e501e78b53a1cda2b96502f2d7197edce56424ab1b29889"}
Nov 03 20:46:38.153503 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.153434    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4" event={"ID":"830b16e0-459d-444c-b7da-c8d6d8576d4f","Type":"ContainerDied","Data":"74ec5585995d88ac5403097f7f672fb5a24e2f1d00dddac3994c531e452080aa"}
Nov 03 20:46:38.153560 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.153518    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"
Nov 03 20:46:38.167407 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.167388    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:46:38.170630 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.170613    4421 scope.go:122] "RemoveContainer" containerID="57d92c1a1e5f97709e501e78b53a1cda2b96502f2d7197edce56424ab1b29889"
Nov 03 20:46:38.183364 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.183334    4421 scope.go:122] "RemoveContainer" containerID="296173dacd6b5ac0ad08d8f5cb4c6356ebbde4afc2a09ebf492e25eb42971f2b"
Nov 03 20:46:38.184106 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.184087    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"]
Nov 03 20:46:38.190921 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.190895    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-7rvj4"]
Nov 03 20:46:38.194118 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.194103    4421 scope.go:122] "RemoveContainer" containerID="57d92c1a1e5f97709e501e78b53a1cda2b96502f2d7197edce56424ab1b29889"
Nov 03 20:46:38.194369 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:38.194348    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57d92c1a1e5f97709e501e78b53a1cda2b96502f2d7197edce56424ab1b29889\": container with ID starting with 57d92c1a1e5f97709e501e78b53a1cda2b96502f2d7197edce56424ab1b29889 not found: ID does not exist" containerID="57d92c1a1e5f97709e501e78b53a1cda2b96502f2d7197edce56424ab1b29889"
Nov 03 20:46:38.194425 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.194379    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57d92c1a1e5f97709e501e78b53a1cda2b96502f2d7197edce56424ab1b29889"} err="failed to get container status \"57d92c1a1e5f97709e501e78b53a1cda2b96502f2d7197edce56424ab1b29889\": rpc error: code = NotFound desc = could not find container \"57d92c1a1e5f97709e501e78b53a1cda2b96502f2d7197edce56424ab1b29889\": container with ID starting with 57d92c1a1e5f97709e501e78b53a1cda2b96502f2d7197edce56424ab1b29889 not found: ID does not exist"
Nov 03 20:46:38.194425 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.194396    4421 scope.go:122] "RemoveContainer" containerID="296173dacd6b5ac0ad08d8f5cb4c6356ebbde4afc2a09ebf492e25eb42971f2b"
Nov 03 20:46:38.194687 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:38.194669    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"296173dacd6b5ac0ad08d8f5cb4c6356ebbde4afc2a09ebf492e25eb42971f2b\": container with ID starting with 296173dacd6b5ac0ad08d8f5cb4c6356ebbde4afc2a09ebf492e25eb42971f2b not found: ID does not exist" containerID="296173dacd6b5ac0ad08d8f5cb4c6356ebbde4afc2a09ebf492e25eb42971f2b"
Nov 03 20:46:38.194748 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.194696    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"296173dacd6b5ac0ad08d8f5cb4c6356ebbde4afc2a09ebf492e25eb42971f2b"} err="failed to get container status \"296173dacd6b5ac0ad08d8f5cb4c6356ebbde4afc2a09ebf492e25eb42971f2b\": rpc error: code = NotFound desc = could not find container \"296173dacd6b5ac0ad08d8f5cb4c6356ebbde4afc2a09ebf492e25eb42971f2b\": container with ID starting with 296173dacd6b5ac0ad08d8f5cb4c6356ebbde4afc2a09ebf492e25eb42971f2b not found: ID does not exist"
Nov 03 20:46:38.245383 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.245339    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/317e6b68-68f8-4220-a3ef-2f0999a19ad9-kube-api-access-msp7k\" (UniqueName: \"kubernetes.io/projected/317e6b68-68f8-4220-a3ef-2f0999a19ad9-kube-api-access-msp7k\") pod \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") "
Nov 03 20:46:38.245639 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.245437    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-envoy\") pod \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") "
Nov 03 20:46:38.245639 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.245495    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-token\" (UniqueName: \"kubernetes.io/projected/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-token\") pod \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") "
Nov 03 20:46:38.245639 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.245563    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-workload-certs\") pod \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") "
Nov 03 20:46:38.245921 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.245714    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istiod-ca-cert\") pod \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") "
Nov 03 20:46:38.245921 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.245764    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-credential-socket\") pod \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") "
Nov 03 20:46:38.245921 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.245876    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-podinfo\") pod \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") "
Nov 03 20:46:38.245921 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.245914    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-data\") pod \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") "
Nov 03 20:46:38.246270 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.245997    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-workload-certs" pod "317e6b68-68f8-4220-a3ef-2f0999a19ad9" (UID: "317e6b68-68f8-4220-a3ef-2f0999a19ad9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:38.246270 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.246048    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-workload-socket\") pod \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\" (UID: \"317e6b68-68f8-4220-a3ef-2f0999a19ad9\") "
Nov 03 20:46:38.247024 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.246979    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:38.247703 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.247675    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istiod-ca-cert" pod "317e6b68-68f8-4220-a3ef-2f0999a19ad9" (UID: "317e6b68-68f8-4220-a3ef-2f0999a19ad9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:46:38.248061 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.248039    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-data" pod "317e6b68-68f8-4220-a3ef-2f0999a19ad9" (UID: "317e6b68-68f8-4220-a3ef-2f0999a19ad9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:38.248661 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.248635    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/317e6b68-68f8-4220-a3ef-2f0999a19ad9-kube-api-access-msp7k" pod "317e6b68-68f8-4220-a3ef-2f0999a19ad9" (UID: "317e6b68-68f8-4220-a3ef-2f0999a19ad9"). InnerVolumeSpecName "kube-api-access-msp7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:46:38.248761 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.248709    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-envoy" pod "317e6b68-68f8-4220-a3ef-2f0999a19ad9" (UID: "317e6b68-68f8-4220-a3ef-2f0999a19ad9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:38.249305 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.249279    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-credential-socket" pod "317e6b68-68f8-4220-a3ef-2f0999a19ad9" (UID: "317e6b68-68f8-4220-a3ef-2f0999a19ad9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:38.249410 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.249342    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-workload-socket" pod "317e6b68-68f8-4220-a3ef-2f0999a19ad9" (UID: "317e6b68-68f8-4220-a3ef-2f0999a19ad9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:38.249520 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.249498    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-token" pod "317e6b68-68f8-4220-a3ef-2f0999a19ad9" (UID: "317e6b68-68f8-4220-a3ef-2f0999a19ad9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:46:38.251005 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.250839    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-podinfo" pod "317e6b68-68f8-4220-a3ef-2f0999a19ad9" (UID: "317e6b68-68f8-4220-a3ef-2f0999a19ad9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:46:38.348759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.348475    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:38.348759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.348509    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:38.348759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.348525    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:38.348759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.348538    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msp7k\" (UniqueName: \"kubernetes.io/projected/317e6b68-68f8-4220-a3ef-2f0999a19ad9-kube-api-access-msp7k\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:38.348759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.348554    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:38.348759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.348570    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:38.348759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.348583    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/317e6b68-68f8-4220-a3ef-2f0999a19ad9-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:38.348759 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:38.348642    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/317e6b68-68f8-4220-a3ef-2f0999a19ad9-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:39.164442 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:39.164410    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw" event={"ID":"317e6b68-68f8-4220-a3ef-2f0999a19ad9","Type":"ContainerDied","Data":"c8e1e37513ed3875cc695de89ef18758eac04f97f0722c92aacf025e6a42809a"}
Nov 03 20:46:39.164958 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:39.164453    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"
Nov 03 20:46:39.164958 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:39.164475    4421 scope.go:122] "RemoveContainer" containerID="2b14fce9a07d8e0822f9cb6964c3462892594276505a91b7dc0a9ff28e28e167"
Nov 03 20:46:39.218642 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:39.214898    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"]
Nov 03 20:46:39.221135 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:39.221105    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-x8tpw"]
Nov 03 20:46:40.020545 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:40.020512    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="317e6b68-68f8-4220-a3ef-2f0999a19ad9" path="/var/lib/kubelet/pods/317e6b68-68f8-4220-a3ef-2f0999a19ad9/volumes"
Nov 03 20:46:40.021197 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:40.021179    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="830b16e0-459d-444c-b7da-c8d6d8576d4f" path="/var/lib/kubelet/pods/830b16e0-459d-444c-b7da-c8d6d8576d4f/volumes"
Nov 03 20:46:46.186005 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.185971    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85"]
Nov 03 20:46:46.186869 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.186853    4421 cpu_manager.go:414] "containerMap: removing container" podUID="317e6b68-68f8-4220-a3ef-2f0999a19ad9" containerName="istio-proxy"
Nov 03 20:46:46.186942 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.186870    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="317e6b68-68f8-4220-a3ef-2f0999a19ad9" containerName="istio-proxy"
Nov 03 20:46:46.186942 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.186884    4421 cpu_manager.go:414] "containerMap: removing container" podUID="830b16e0-459d-444c-b7da-c8d6d8576d4f" containerName="istio-proxy"
Nov 03 20:46:46.186942 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.186888    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="830b16e0-459d-444c-b7da-c8d6d8576d4f" containerName="istio-proxy"
Nov 03 20:46:46.186942 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.186899    4421 cpu_manager.go:414] "containerMap: removing container" podUID="3cf17e0a-2b24-4633-afb7-44eebab411d3" containerName="istio-proxy"
Nov 03 20:46:46.186942 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.186904    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cf17e0a-2b24-4633-afb7-44eebab411d3" containerName="istio-proxy"
Nov 03 20:46:46.186942 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.186929    4421 cpu_manager.go:414] "containerMap: removing container" podUID="830b16e0-459d-444c-b7da-c8d6d8576d4f" containerName="istio-proxy"
Nov 03 20:46:46.186942 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.186935    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="830b16e0-459d-444c-b7da-c8d6d8576d4f" containerName="istio-proxy"
Nov 03 20:46:46.186942 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.186945    4421 cpu_manager.go:414] "containerMap: removing container" podUID="317e6b68-68f8-4220-a3ef-2f0999a19ad9" containerName="istio-proxy"
Nov 03 20:46:46.187195 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.186950    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="317e6b68-68f8-4220-a3ef-2f0999a19ad9" containerName="istio-proxy"
Nov 03 20:46:46.187195 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.187086    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="317e6b68-68f8-4220-a3ef-2f0999a19ad9" containerName="istio-proxy"
Nov 03 20:46:46.187195 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.187093    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="830b16e0-459d-444c-b7da-c8d6d8576d4f" containerName="istio-proxy"
Nov 03 20:46:46.187195 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.187098    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cf17e0a-2b24-4633-afb7-44eebab411d3" containerName="istio-proxy"
Nov 03 20:46:46.187195 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.187110    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="830b16e0-459d-444c-b7da-c8d6d8576d4f" containerName="istio-proxy"
Nov 03 20:46:46.193160 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.193140    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85"
Nov 03 20:46:46.198994 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.198966    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"kube-root-ca.crt\""
Nov 03 20:46:46.199141 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.198996    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"routing-overlay\""
Nov 03 20:46:46.199141 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.198996    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"payload-processing-external-model-plugins-e2e-extmod-a-37aa95\""
Nov 03 20:46:46.199141 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.199016    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"agc-cred-233866678b02\""
Nov 03 20:46:46.199141 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.199034    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"openshift-service-ca.crt\""
Nov 03 20:46:46.199141 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.199073    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-37aa95\"/\"payload-processing-external-model-e2e--0dce6837-dockercfg-8c4ts\""
Nov 03 20:46:46.199879 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.199858    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85"]
Nov 03 20:46:46.260510 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.260476    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4db08cf7-f986-41e1-8a22-61080f6a7576-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85\" (UID: \"4db08cf7-f986-41e1-8a22-61080f6a7576\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85"
Nov 03 20:46:46.260758 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.260524    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4db08cf7-f986-41e1-8a22-61080f6a7576-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85\" (UID: \"4db08cf7-f986-41e1-8a22-61080f6a7576\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85"
Nov 03 20:46:46.260916 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.260890    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4db08cf7-f986-41e1-8a22-61080f6a7576-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85\" (UID: \"4db08cf7-f986-41e1-8a22-61080f6a7576\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85"
Nov 03 20:46:46.261014 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.260986    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgts7\" (UniqueName: \"kubernetes.io/projected/4db08cf7-f986-41e1-8a22-61080f6a7576-kube-api-access-zgts7\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85\" (UID: \"4db08cf7-f986-41e1-8a22-61080f6a7576\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85"
Nov 03 20:46:46.362717 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.362688    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4db08cf7-f986-41e1-8a22-61080f6a7576-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85\" (UID: \"4db08cf7-f986-41e1-8a22-61080f6a7576\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85"
Nov 03 20:46:46.362973 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.362755    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgts7\" (UniqueName: \"kubernetes.io/projected/4db08cf7-f986-41e1-8a22-61080f6a7576-kube-api-access-zgts7\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85\" (UID: \"4db08cf7-f986-41e1-8a22-61080f6a7576\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85"
Nov 03 20:46:46.362973 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.362846    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4db08cf7-f986-41e1-8a22-61080f6a7576-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85\" (UID: \"4db08cf7-f986-41e1-8a22-61080f6a7576\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85"
Nov 03 20:46:46.362973 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.362877    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4db08cf7-f986-41e1-8a22-61080f6a7576-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85\" (UID: \"4db08cf7-f986-41e1-8a22-61080f6a7576\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85"
Nov 03 20:46:46.363503 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.363479    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4db08cf7-f986-41e1-8a22-61080f6a7576-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85\" (UID: \"4db08cf7-f986-41e1-8a22-61080f6a7576\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85"
Nov 03 20:46:46.363687 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.363662    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4db08cf7-f986-41e1-8a22-61080f6a7576-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85\" (UID: \"4db08cf7-f986-41e1-8a22-61080f6a7576\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85"
Nov 03 20:46:46.365733 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.365711    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4db08cf7-f986-41e1-8a22-61080f6a7576-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85\" (UID: \"4db08cf7-f986-41e1-8a22-61080f6a7576\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85"
Nov 03 20:46:46.375385 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.375364    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgts7\" (UniqueName: \"kubernetes.io/projected/4db08cf7-f986-41e1-8a22-61080f6a7576-kube-api-access-zgts7\") pod \"payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85\" (UID: \"4db08cf7-f986-41e1-8a22-61080f6a7576\") " pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85"
Nov 03 20:46:46.451190 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.451151    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g"]
Nov 03 20:46:46.452915 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.452887    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="317e6b68-68f8-4220-a3ef-2f0999a19ad9" containerName="istio-proxy"
Nov 03 20:46:46.465523 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.465500    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g"]
Nov 03 20:46:46.465708 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.465615    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g"
Nov 03 20:46:46.469355 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.469319    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c7aecb\"/\"agc-cred-65165f0977f9\""
Nov 03 20:46:46.469533 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.469509    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c7aecb\"/\"payload-processing-external-model-e2e--37db5ca3-dockercfg-d97nx\""
Nov 03 20:46:46.469896 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.469871    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c7aecb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-c7aecb\""
Nov 03 20:46:46.470034 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.469882    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c7aecb\"/\"openshift-service-ca.crt\""
Nov 03 20:46:46.470571 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.470545    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c7aecb\"/\"kube-root-ca.crt\""
Nov 03 20:46:46.470733 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.470651    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c7aecb\"/\"routing-overlay\""
Nov 03 20:46:46.514404 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.514378    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85"
Nov 03 20:46:46.578679 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.574179    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gdzt\" (UniqueName: \"kubernetes.io/projected/85b7ad71-e155-481e-8032-8198db21ce7a-kube-api-access-9gdzt\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g\" (UID: \"85b7ad71-e155-481e-8032-8198db21ce7a\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g"
Nov 03 20:46:46.578679 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.574317    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85b7ad71-e155-481e-8032-8198db21ce7a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g\" (UID: \"85b7ad71-e155-481e-8032-8198db21ce7a\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g"
Nov 03 20:46:46.578679 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.574348    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85b7ad71-e155-481e-8032-8198db21ce7a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g\" (UID: \"85b7ad71-e155-481e-8032-8198db21ce7a\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g"
Nov 03 20:46:46.578679 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.574517    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b7ad71-e155-481e-8032-8198db21ce7a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g\" (UID: \"85b7ad71-e155-481e-8032-8198db21ce7a\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g"
Nov 03 20:46:46.677197 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.676928    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gdzt\" (UniqueName: \"kubernetes.io/projected/85b7ad71-e155-481e-8032-8198db21ce7a-kube-api-access-9gdzt\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g\" (UID: \"85b7ad71-e155-481e-8032-8198db21ce7a\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g"
Nov 03 20:46:46.677197 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.677086    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85b7ad71-e155-481e-8032-8198db21ce7a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g\" (UID: \"85b7ad71-e155-481e-8032-8198db21ce7a\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g"
Nov 03 20:46:46.677197 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.677128    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85b7ad71-e155-481e-8032-8198db21ce7a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g\" (UID: \"85b7ad71-e155-481e-8032-8198db21ce7a\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g"
Nov 03 20:46:46.677512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.677345    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b7ad71-e155-481e-8032-8198db21ce7a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g\" (UID: \"85b7ad71-e155-481e-8032-8198db21ce7a\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g"
Nov 03 20:46:46.679116 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.678696    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b7ad71-e155-481e-8032-8198db21ce7a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g\" (UID: \"85b7ad71-e155-481e-8032-8198db21ce7a\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g"
Nov 03 20:46:46.679116 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.679031    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85b7ad71-e155-481e-8032-8198db21ce7a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g\" (UID: \"85b7ad71-e155-481e-8032-8198db21ce7a\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g"
Nov 03 20:46:46.684261 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.684239    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85b7ad71-e155-481e-8032-8198db21ce7a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g\" (UID: \"85b7ad71-e155-481e-8032-8198db21ce7a\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g"
Nov 03 20:46:46.691551 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.691528    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gdzt\" (UniqueName: \"kubernetes.io/projected/85b7ad71-e155-481e-8032-8198db21ce7a-kube-api-access-9gdzt\") pod \"payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g\" (UID: \"85b7ad71-e155-481e-8032-8198db21ce7a\") " pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g"
Nov 03 20:46:46.790052 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.789958    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g"
Nov 03 20:46:46.873019 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:46:46.872983    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db08cf7_f986_41e1_8a22_61080f6a7576.slice/crio-d11bf829039fb3afe6464dda76f778c087f73e092a40d038f67d1540f889ed4a WatchSource:0}: Error finding container d11bf829039fb3afe6464dda76f778c087f73e092a40d038f67d1540f889ed4a: Status 404 returned error can't find the container with id d11bf829039fb3afe6464dda76f778c087f73e092a40d038f67d1540f889ed4a
Nov 03 20:46:46.874404 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:46.874368    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85"]
Nov 03 20:46:47.085493 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:47.085461    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g"]
Nov 03 20:46:47.085647 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:46:47.085533    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b7ad71_e155_481e_8032_8198db21ce7a.slice/crio-6d6188394e2e249190ea7349a9c105cd68324cd2a6cb76a97b11e5e8160fcd0c WatchSource:0}: Error finding container 6d6188394e2e249190ea7349a9c105cd68324cd2a6cb76a97b11e5e8160fcd0c: Status 404 returned error can't find the container with id 6d6188394e2e249190ea7349a9c105cd68324cd2a6cb76a97b11e5e8160fcd0c
Nov 03 20:46:47.238613 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:47.238559    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g" event={"ID":"85b7ad71-e155-481e-8032-8198db21ce7a","Type":"ContainerStarted","Data":"28effb61d4c044d17e724d4e170cabb9a21f901594ff89302e2282ede4dbc1c4"}
Nov 03 20:46:47.239079 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:47.238676    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g"
Nov 03 20:46:47.239079 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:47.238705    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g" event={"ID":"85b7ad71-e155-481e-8032-8198db21ce7a","Type":"ContainerStarted","Data":"6d6188394e2e249190ea7349a9c105cd68324cd2a6cb76a97b11e5e8160fcd0c"}
Nov 03 20:46:47.240665 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:47.240629    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85" event={"ID":"4db08cf7-f986-41e1-8a22-61080f6a7576","Type":"ContainerStarted","Data":"c2b99ae9368cfd5fd289c9648bffda35b5c9b2c3f017f92c13a38cf64d5aaacc"}
Nov 03 20:46:47.240810 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:47.240681    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85" event={"ID":"4db08cf7-f986-41e1-8a22-61080f6a7576","Type":"ContainerStarted","Data":"d11bf829039fb3afe6464dda76f778c087f73e092a40d038f67d1540f889ed4a"}
Nov 03 20:46:47.240810 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:47.240711    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85"
Nov 03 20:46:47.258863 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:47.258815    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g" podStartSLOduration=1.258803079 podStartE2EDuration="1.258803079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:47.257062707 +0000 UTC m=+2737.390191170" watchObservedRunningTime="2026-11-03 20:46:47.258803079 +0000 UTC m=+2737.391931542"
Nov 03 20:46:47.278053 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:47.277832    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85" podStartSLOduration=1.277808842 podStartE2EDuration="1.277808842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:47.276989778 +0000 UTC m=+2737.410118251" watchObservedRunningTime="2026-11-03 20:46:47.277808842 +0000 UTC m=+2737.410937305"
Nov 03 20:46:47.807098 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:47.806209    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g"]
Nov 03 20:46:48.340174 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:48.333542    4421 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g" secret="" err="secret \"payload-processing-external-model-e2e--37db5ca3-dockercfg-d97nx\" not found"
Nov 03 20:46:48.417806 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:48.417774    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85"]
Nov 03 20:46:48.438757 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:48.438723    4421 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-plugins-e2e-extmod-b-c7aecb: configmap "payload-processing-external-model-plugins-e2e-extmod-b-c7aecb" not found
Nov 03 20:46:48.438962 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:48.438828    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85b7ad71-e155-481e-8032-8198db21ce7a-plugins-config-volume podName:85b7ad71-e155-481e-8032-8198db21ce7a nodeName:}" failed. No retries permitted until 2026-11-03 20:46:48.938807546 +0000 UTC m=+2739.071935988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/85b7ad71-e155-481e-8032-8198db21ce7a-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g" (UID: "85b7ad71-e155-481e-8032-8198db21ce7a") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-c7aecb" not found
Nov 03 20:46:48.439045 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:48.439028    4421 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-plugins-e2e-extmod-a-37aa95: configmap "payload-processing-external-model-plugins-e2e-extmod-a-37aa95" not found
Nov 03 20:46:48.439105 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:48.439078    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4db08cf7-f986-41e1-8a22-61080f6a7576-plugins-config-volume podName:4db08cf7-f986-41e1-8a22-61080f6a7576 nodeName:}" failed. No retries permitted until 2026-11-03 20:46:48.939067372 +0000 UTC m=+2739.072195811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/4db08cf7-f986-41e1-8a22-61080f6a7576-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85" (UID: "4db08cf7-f986-41e1-8a22-61080f6a7576") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-37aa95" not found
Nov 03 20:46:48.440226 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:48.440203    4421 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c7aecb/agc-cred-65165f0977f9: secret "agc-cred-65165f0977f9" not found
Nov 03 20:46:48.440226 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:48.440223    4421 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g: secret "agc-cred-65165f0977f9" not found
Nov 03 20:46:48.440418 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:48.440261    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85b7ad71-e155-481e-8032-8198db21ce7a-provider-credentials podName:85b7ad71-e155-481e-8032-8198db21ce7a nodeName:}" failed. No retries permitted until 2026-11-03 20:46:48.940250571 +0000 UTC m=+2739.073379025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/85b7ad71-e155-481e-8032-8198db21ce7a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g" (UID: "85b7ad71-e155-481e-8032-8198db21ce7a") : secret "agc-cred-65165f0977f9" not found
Nov 03 20:46:48.440418 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:48.440308    4421 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-37aa95/agc-cred-233866678b02: secret "agc-cred-233866678b02" not found
Nov 03 20:46:48.440418 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:48.440314    4421 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85: secret "agc-cred-233866678b02" not found
Nov 03 20:46:48.440418 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:48.440335    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4db08cf7-f986-41e1-8a22-61080f6a7576-provider-credentials podName:4db08cf7-f986-41e1-8a22-61080f6a7576 nodeName:}" failed. No retries permitted until 2026-11-03 20:46:48.940328637 +0000 UTC m=+2739.073457090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4db08cf7-f986-41e1-8a22-61080f6a7576-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85" (UID: "4db08cf7-f986-41e1-8a22-61080f6a7576") : secret "agc-cred-233866678b02" not found
Nov 03 20:46:48.959875 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:48.959845    4421 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-plugins-e2e-extmod-b-c7aecb: configmap "payload-processing-external-model-plugins-e2e-extmod-b-c7aecb" not found
Nov 03 20:46:48.960072 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:48.959923    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85b7ad71-e155-481e-8032-8198db21ce7a-plugins-config-volume podName:85b7ad71-e155-481e-8032-8198db21ce7a nodeName:}" failed. No retries permitted until 2026-11-03 20:46:49.959908632 +0000 UTC m=+2740.093037086 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/85b7ad71-e155-481e-8032-8198db21ce7a-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g" (UID: "85b7ad71-e155-481e-8032-8198db21ce7a") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-c7aecb" not found
Nov 03 20:46:48.960137 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:48.960108    4421 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-plugins-e2e-extmod-a-37aa95: configmap "payload-processing-external-model-plugins-e2e-extmod-a-37aa95" not found
Nov 03 20:46:48.960188 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:48.960175    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4db08cf7-f986-41e1-8a22-61080f6a7576-plugins-config-volume podName:4db08cf7-f986-41e1-8a22-61080f6a7576 nodeName:}" failed. No retries permitted until 2026-11-03 20:46:49.960160252 +0000 UTC m=+2740.093288691 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/4db08cf7-f986-41e1-8a22-61080f6a7576-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85" (UID: "4db08cf7-f986-41e1-8a22-61080f6a7576") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-37aa95" not found
Nov 03 20:46:48.960256 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:48.960242    4421 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c7aecb/agc-cred-65165f0977f9: secret "agc-cred-65165f0977f9" not found
Nov 03 20:46:48.960306 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:48.960259    4421 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g: secret "agc-cred-65165f0977f9" not found
Nov 03 20:46:48.960306 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:48.960295    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85b7ad71-e155-481e-8032-8198db21ce7a-provider-credentials podName:85b7ad71-e155-481e-8032-8198db21ce7a nodeName:}" failed. No retries permitted until 2026-11-03 20:46:49.960284625 +0000 UTC m=+2740.093413079 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/85b7ad71-e155-481e-8032-8198db21ce7a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g" (UID: "85b7ad71-e155-481e-8032-8198db21ce7a") : secret "agc-cred-65165f0977f9" not found
Nov 03 20:46:48.960409 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:48.960396    4421 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-37aa95/agc-cred-233866678b02: secret "agc-cred-233866678b02" not found
Nov 03 20:46:48.960456 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:48.960410    4421 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85: secret "agc-cred-233866678b02" not found
Nov 03 20:46:48.960456 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:48.960442    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4db08cf7-f986-41e1-8a22-61080f6a7576-provider-credentials podName:4db08cf7-f986-41e1-8a22-61080f6a7576 nodeName:}" failed. No retries permitted until 2026-11-03 20:46:49.960433708 +0000 UTC m=+2740.093562148 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4db08cf7-f986-41e1-8a22-61080f6a7576-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85" (UID: "4db08cf7-f986-41e1-8a22-61080f6a7576") : secret "agc-cred-233866678b02" not found
Nov 03 20:46:49.632820 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:49.632778    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:46:49.977858 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:49.977826    4421 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-plugins-e2e-extmod-a-37aa95: configmap "payload-processing-external-model-plugins-e2e-extmod-a-37aa95" not found
Nov 03 20:46:49.978061 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:49.977905    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4db08cf7-f986-41e1-8a22-61080f6a7576-plugins-config-volume podName:4db08cf7-f986-41e1-8a22-61080f6a7576 nodeName:}" failed. No retries permitted until 2026-11-03 20:46:51.977885023 +0000 UTC m=+2742.111013463 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/4db08cf7-f986-41e1-8a22-61080f6a7576-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85" (UID: "4db08cf7-f986-41e1-8a22-61080f6a7576") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-37aa95" not found
Nov 03 20:46:49.978061 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:49.977926    4421 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-plugins-e2e-extmod-b-c7aecb: configmap "payload-processing-external-model-plugins-e2e-extmod-b-c7aecb" not found
Nov 03 20:46:49.978061 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:49.977979    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85b7ad71-e155-481e-8032-8198db21ce7a-plugins-config-volume podName:85b7ad71-e155-481e-8032-8198db21ce7a nodeName:}" failed. No retries permitted until 2026-11-03 20:46:51.97796788 +0000 UTC m=+2742.111096350 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/85b7ad71-e155-481e-8032-8198db21ce7a-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g" (UID: "85b7ad71-e155-481e-8032-8198db21ce7a") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-c7aecb" not found
Nov 03 20:46:49.978061 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:49.977932    4421 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c7aecb/agc-cred-65165f0977f9: secret "agc-cred-65165f0977f9" not found
Nov 03 20:46:49.978061 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:49.977998    4421 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g: secret "agc-cred-65165f0977f9" not found
Nov 03 20:46:49.978061 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:49.978002    4421 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-37aa95/agc-cred-233866678b02: secret "agc-cred-233866678b02" not found
Nov 03 20:46:49.978061 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:49.978021    4421 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85: secret "agc-cred-233866678b02" not found
Nov 03 20:46:49.978061 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:49.978024    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85b7ad71-e155-481e-8032-8198db21ce7a-provider-credentials podName:85b7ad71-e155-481e-8032-8198db21ce7a nodeName:}" failed. No retries permitted until 2026-11-03 20:46:51.978017806 +0000 UTC m=+2742.111146246 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/85b7ad71-e155-481e-8032-8198db21ce7a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g" (UID: "85b7ad71-e155-481e-8032-8198db21ce7a") : secret "agc-cred-65165f0977f9" not found
Nov 03 20:46:49.978458 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:49.978080    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4db08cf7-f986-41e1-8a22-61080f6a7576-provider-credentials podName:4db08cf7-f986-41e1-8a22-61080f6a7576 nodeName:}" failed. No retries permitted until 2026-11-03 20:46:51.978069016 +0000 UTC m=+2742.111197455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4db08cf7-f986-41e1-8a22-61080f6a7576-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85" (UID: "4db08cf7-f986-41e1-8a22-61080f6a7576") : secret "agc-cred-233866678b02" not found
Nov 03 20:46:52.011074 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:52.011041    4421 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-plugins-e2e-extmod-a-37aa95: configmap "payload-processing-external-model-plugins-e2e-extmod-a-37aa95" not found
Nov 03 20:46:52.011579 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:52.011116    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4db08cf7-f986-41e1-8a22-61080f6a7576-plugins-config-volume podName:4db08cf7-f986-41e1-8a22-61080f6a7576 nodeName:}" failed. No retries permitted until 2026-11-03 20:46:56.011098296 +0000 UTC m=+2746.144226736 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/4db08cf7-f986-41e1-8a22-61080f6a7576-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85" (UID: "4db08cf7-f986-41e1-8a22-61080f6a7576") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-37aa95" not found
Nov 03 20:46:52.011579 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:52.011169    4421 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-37aa95/agc-cred-233866678b02: secret "agc-cred-233866678b02" not found
Nov 03 20:46:52.011579 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:52.011182    4421 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85: secret "agc-cred-233866678b02" not found
Nov 03 20:46:52.011579 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:52.011208    4421 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-c7aecb/agc-cred-65165f0977f9: secret "agc-cred-65165f0977f9" not found
Nov 03 20:46:52.011579 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:52.011228    4421 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g: secret "agc-cred-65165f0977f9" not found
Nov 03 20:46:52.011579 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:52.011238    4421 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-plugins-e2e-extmod-b-c7aecb: configmap "payload-processing-external-model-plugins-e2e-extmod-b-c7aecb" not found
Nov 03 20:46:52.011579 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:52.011220    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4db08cf7-f986-41e1-8a22-61080f6a7576-provider-credentials podName:4db08cf7-f986-41e1-8a22-61080f6a7576 nodeName:}" failed. No retries permitted until 2026-11-03 20:46:56.011208762 +0000 UTC m=+2746.144337217 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4db08cf7-f986-41e1-8a22-61080f6a7576-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85" (UID: "4db08cf7-f986-41e1-8a22-61080f6a7576") : secret "agc-cred-233866678b02" not found
Nov 03 20:46:52.011579 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:52.011280    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85b7ad71-e155-481e-8032-8198db21ce7a-provider-credentials podName:85b7ad71-e155-481e-8032-8198db21ce7a nodeName:}" failed. No retries permitted until 2026-11-03 20:46:56.011267765 +0000 UTC m=+2746.144396208 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/85b7ad71-e155-481e-8032-8198db21ce7a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g" (UID: "85b7ad71-e155-481e-8032-8198db21ce7a") : secret "agc-cred-65165f0977f9" not found
Nov 03 20:46:52.011579 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:52.011297    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85b7ad71-e155-481e-8032-8198db21ce7a-plugins-config-volume podName:85b7ad71-e155-481e-8032-8198db21ce7a nodeName:}" failed. No retries permitted until 2026-11-03 20:46:56.011289456 +0000 UTC m=+2746.144417896 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/85b7ad71-e155-481e-8032-8198db21ce7a-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g" (UID: "85b7ad71-e155-481e-8032-8198db21ce7a") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-c7aecb" not found
Nov 03 20:46:54.335311 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.335263    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85" podUID="4db08cf7-f986-41e1-8a22-61080f6a7576" containerName="payload-processing" containerID="cri-o://c2b99ae9368cfd5fd289c9648bffda35b5c9b2c3f017f92c13a38cf64d5aaacc" gracePeriod=25
Nov 03 20:46:54.335774 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.335402    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g" podUID="85b7ad71-e155-481e-8032-8198db21ce7a" containerName="payload-processing" containerID="cri-o://28effb61d4c044d17e724d4e170cabb9a21f901594ff89302e2282ede4dbc1c4" gracePeriod=25
Nov 03 20:46:54.680360 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.680334    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g"
Nov 03 20:46:54.683895 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.683875    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85"
Nov 03 20:46:54.777052 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.777020    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b7ad71-e155-481e-8032-8198db21ce7a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/85b7ad71-e155-481e-8032-8198db21ce7a-provider-credentials\") pod \"85b7ad71-e155-481e-8032-8198db21ce7a\" (UID: \"85b7ad71-e155-481e-8032-8198db21ce7a\") "
Nov 03 20:46:54.777215 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.777141    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b7ad71-e155-481e-8032-8198db21ce7a-kube-api-access-9gdzt\" (UniqueName: \"kubernetes.io/projected/85b7ad71-e155-481e-8032-8198db21ce7a-kube-api-access-9gdzt\") pod \"85b7ad71-e155-481e-8032-8198db21ce7a\" (UID: \"85b7ad71-e155-481e-8032-8198db21ce7a\") "
Nov 03 20:46:54.777215 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.777164    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85b7ad71-e155-481e-8032-8198db21ce7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b7ad71-e155-481e-8032-8198db21ce7a-plugins-config-volume\") pod \"85b7ad71-e155-481e-8032-8198db21ce7a\" (UID: \"85b7ad71-e155-481e-8032-8198db21ce7a\") "
Nov 03 20:46:54.777302 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.777216    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85b7ad71-e155-481e-8032-8198db21ce7a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85b7ad71-e155-481e-8032-8198db21ce7a-routing-overlay-volume\") pod \"85b7ad71-e155-481e-8032-8198db21ce7a\" (UID: \"85b7ad71-e155-481e-8032-8198db21ce7a\") "
Nov 03 20:46:54.777564 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.777536    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85b7ad71-e155-481e-8032-8198db21ce7a-routing-overlay-volume" pod "85b7ad71-e155-481e-8032-8198db21ce7a" (UID: "85b7ad71-e155-481e-8032-8198db21ce7a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:46:54.777698 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.777582    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85b7ad71-e155-481e-8032-8198db21ce7a-plugins-config-volume" pod "85b7ad71-e155-481e-8032-8198db21ce7a" (UID: "85b7ad71-e155-481e-8032-8198db21ce7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:46:54.777967 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.777949    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85b7ad71-e155-481e-8032-8198db21ce7a-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:54.778040 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.777971    4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/85b7ad71-e155-481e-8032-8198db21ce7a-routing-overlay-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:54.779554 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.779530    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b7ad71-e155-481e-8032-8198db21ce7a-kube-api-access-9gdzt" pod "85b7ad71-e155-481e-8032-8198db21ce7a" (UID: "85b7ad71-e155-481e-8032-8198db21ce7a"). InnerVolumeSpecName "kube-api-access-9gdzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:46:54.779682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.779557    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b7ad71-e155-481e-8032-8198db21ce7a-provider-credentials" pod "85b7ad71-e155-481e-8032-8198db21ce7a" (UID: "85b7ad71-e155-481e-8032-8198db21ce7a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:46:54.879130 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.879045    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db08cf7-f986-41e1-8a22-61080f6a7576-kube-api-access-zgts7\" (UniqueName: \"kubernetes.io/projected/4db08cf7-f986-41e1-8a22-61080f6a7576-kube-api-access-zgts7\") pod \"4db08cf7-f986-41e1-8a22-61080f6a7576\" (UID: \"4db08cf7-f986-41e1-8a22-61080f6a7576\") "
Nov 03 20:46:54.879285 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.879133    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db08cf7-f986-41e1-8a22-61080f6a7576-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4db08cf7-f986-41e1-8a22-61080f6a7576-provider-credentials\") pod \"4db08cf7-f986-41e1-8a22-61080f6a7576\" (UID: \"4db08cf7-f986-41e1-8a22-61080f6a7576\") "
Nov 03 20:46:54.879285 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.879203    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4db08cf7-f986-41e1-8a22-61080f6a7576-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4db08cf7-f986-41e1-8a22-61080f6a7576-routing-overlay-volume\") pod \"4db08cf7-f986-41e1-8a22-61080f6a7576\" (UID: \"4db08cf7-f986-41e1-8a22-61080f6a7576\") "
Nov 03 20:46:54.879419 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.879400    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4db08cf7-f986-41e1-8a22-61080f6a7576-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4db08cf7-f986-41e1-8a22-61080f6a7576-plugins-config-volume\") pod \"4db08cf7-f986-41e1-8a22-61080f6a7576\" (UID: \"4db08cf7-f986-41e1-8a22-61080f6a7576\") "
Nov 03 20:46:54.879638 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.879582    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db08cf7-f986-41e1-8a22-61080f6a7576-routing-overlay-volume" pod "4db08cf7-f986-41e1-8a22-61080f6a7576" (UID: "4db08cf7-f986-41e1-8a22-61080f6a7576"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:46:54.879788 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.879698    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db08cf7-f986-41e1-8a22-61080f6a7576-plugins-config-volume" pod "4db08cf7-f986-41e1-8a22-61080f6a7576" (UID: "4db08cf7-f986-41e1-8a22-61080f6a7576"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:46:54.880255 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.880234    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4db08cf7-f986-41e1-8a22-61080f6a7576-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:54.880358 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.880260    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gdzt\" (UniqueName: \"kubernetes.io/projected/85b7ad71-e155-481e-8032-8198db21ce7a-kube-api-access-9gdzt\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:54.880358 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.880274    4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4db08cf7-f986-41e1-8a22-61080f6a7576-routing-overlay-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:54.880358 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.880287    4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/85b7ad71-e155-481e-8032-8198db21ce7a-provider-credentials\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:54.881659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.881634    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db08cf7-f986-41e1-8a22-61080f6a7576-provider-credentials" pod "4db08cf7-f986-41e1-8a22-61080f6a7576" (UID: "4db08cf7-f986-41e1-8a22-61080f6a7576"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:46:54.881659 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.881721    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db08cf7-f986-41e1-8a22-61080f6a7576-kube-api-access-zgts7" pod "4db08cf7-f986-41e1-8a22-61080f6a7576" (UID: "4db08cf7-f986-41e1-8a22-61080f6a7576"). InnerVolumeSpecName "kube-api-access-zgts7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:46:54.982105 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.982072    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgts7\" (UniqueName: \"kubernetes.io/projected/4db08cf7-f986-41e1-8a22-61080f6a7576-kube-api-access-zgts7\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:54.982105 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:54.982099    4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4db08cf7-f986-41e1-8a22-61080f6a7576-provider-credentials\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:55.386259 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:55.386224    4421 generic.go:356] "Generic (PLEG): container finished" podID="4db08cf7-f986-41e1-8a22-61080f6a7576" containerID="c2b99ae9368cfd5fd289c9648bffda35b5c9b2c3f017f92c13a38cf64d5aaacc" exitCode=0
Nov 03 20:46:55.386704 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:55.386283    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85"
Nov 03 20:46:55.386704 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:55.386313    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85" event={"ID":"4db08cf7-f986-41e1-8a22-61080f6a7576","Type":"ContainerDied","Data":"c2b99ae9368cfd5fd289c9648bffda35b5c9b2c3f017f92c13a38cf64d5aaacc"}
Nov 03 20:46:55.386704 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:55.386378    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85" event={"ID":"4db08cf7-f986-41e1-8a22-61080f6a7576","Type":"ContainerDied","Data":"d11bf829039fb3afe6464dda76f778c087f73e092a40d038f67d1540f889ed4a"}
Nov 03 20:46:55.386704 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:55.386404    4421 scope.go:122] "RemoveContainer" containerID="c2b99ae9368cfd5fd289c9648bffda35b5c9b2c3f017f92c13a38cf64d5aaacc"
Nov 03 20:46:55.388695 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:55.388671    4421 generic.go:356] "Generic (PLEG): container finished" podID="85b7ad71-e155-481e-8032-8198db21ce7a" containerID="28effb61d4c044d17e724d4e170cabb9a21f901594ff89302e2282ede4dbc1c4" exitCode=0
Nov 03 20:46:55.388794 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:55.388718    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g"
Nov 03 20:46:55.388794 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:55.388727    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g" event={"ID":"85b7ad71-e155-481e-8032-8198db21ce7a","Type":"ContainerDied","Data":"28effb61d4c044d17e724d4e170cabb9a21f901594ff89302e2282ede4dbc1c4"}
Nov 03 20:46:55.388794 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:55.388772    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g" event={"ID":"85b7ad71-e155-481e-8032-8198db21ce7a","Type":"ContainerDied","Data":"6d6188394e2e249190ea7349a9c105cd68324cd2a6cb76a97b11e5e8160fcd0c"}
Nov 03 20:46:55.398344 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:55.398325    4421 scope.go:122] "RemoveContainer" containerID="c2b99ae9368cfd5fd289c9648bffda35b5c9b2c3f017f92c13a38cf64d5aaacc"
Nov 03 20:46:55.398677 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:55.398653    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2b99ae9368cfd5fd289c9648bffda35b5c9b2c3f017f92c13a38cf64d5aaacc\": container with ID starting with c2b99ae9368cfd5fd289c9648bffda35b5c9b2c3f017f92c13a38cf64d5aaacc not found: ID does not exist" containerID="c2b99ae9368cfd5fd289c9648bffda35b5c9b2c3f017f92c13a38cf64d5aaacc"
Nov 03 20:46:55.398768 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:55.398684    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2b99ae9368cfd5fd289c9648bffda35b5c9b2c3f017f92c13a38cf64d5aaacc"} err="failed to get container status \"c2b99ae9368cfd5fd289c9648bffda35b5c9b2c3f017f92c13a38cf64d5aaacc\": rpc error: code = NotFound desc = could not find container \"c2b99ae9368cfd5fd289c9648bffda35b5c9b2c3f017f92c13a38cf64d5aaacc\": container with ID starting with c2b99ae9368cfd5fd289c9648bffda35b5c9b2c3f017f92c13a38cf64d5aaacc not found: ID does not exist"
Nov 03 20:46:55.398768 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:55.398700    4421 scope.go:122] "RemoveContainer" containerID="28effb61d4c044d17e724d4e170cabb9a21f901594ff89302e2282ede4dbc1c4"
Nov 03 20:46:55.409228 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:55.409213    4421 scope.go:122] "RemoveContainer" containerID="28effb61d4c044d17e724d4e170cabb9a21f901594ff89302e2282ede4dbc1c4"
Nov 03 20:46:55.409521 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:46:55.409496    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28effb61d4c044d17e724d4e170cabb9a21f901594ff89302e2282ede4dbc1c4\": container with ID starting with 28effb61d4c044d17e724d4e170cabb9a21f901594ff89302e2282ede4dbc1c4 not found: ID does not exist" containerID="28effb61d4c044d17e724d4e170cabb9a21f901594ff89302e2282ede4dbc1c4"
Nov 03 20:46:55.409625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:55.409532    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28effb61d4c044d17e724d4e170cabb9a21f901594ff89302e2282ede4dbc1c4"} err="failed to get container status \"28effb61d4c044d17e724d4e170cabb9a21f901594ff89302e2282ede4dbc1c4\": rpc error: code = NotFound desc = could not find container \"28effb61d4c044d17e724d4e170cabb9a21f901594ff89302e2282ede4dbc1c4\": container with ID starting with 28effb61d4c044d17e724d4e170cabb9a21f901594ff89302e2282ede4dbc1c4 not found: ID does not exist"
Nov 03 20:46:55.418149 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:55.418126    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85"]
Nov 03 20:46:55.422573 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:55.422556    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-37aa95/payload-processing-external-model-e2e-extmod-a-37aa95-57fcsxd85"]
Nov 03 20:46:55.433433 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:55.433414    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g"]
Nov 03 20:46:55.437353 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:55.437334    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c7aecb/payload-processing-external-model-e2e-extmod-b-c7aecb-dc96wlc2g"]
Nov 03 20:46:56.028308 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:56.028272    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4db08cf7-f986-41e1-8a22-61080f6a7576" path="/var/lib/kubelet/pods/4db08cf7-f986-41e1-8a22-61080f6a7576/volumes"
Nov 03 20:46:56.029067 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:46:56.029039    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85b7ad71-e155-481e-8032-8198db21ce7a" path="/var/lib/kubelet/pods/85b7ad71-e155-481e-8032-8198db21ce7a/volumes"
Nov 03 20:47:01.418137 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.418107    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj"]
Nov 03 20:47:01.419018 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.419003    4421 cpu_manager.go:414] "containerMap: removing container" podUID="85b7ad71-e155-481e-8032-8198db21ce7a" containerName="payload-processing"
Nov 03 20:47:01.419090 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.419020    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85b7ad71-e155-481e-8032-8198db21ce7a" containerName="payload-processing"
Nov 03 20:47:01.419090 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.419080    4421 cpu_manager.go:414] "containerMap: removing container" podUID="4db08cf7-f986-41e1-8a22-61080f6a7576" containerName="payload-processing"
Nov 03 20:47:01.419090 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.419087    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db08cf7-f986-41e1-8a22-61080f6a7576" containerName="payload-processing"
Nov 03 20:47:01.419242 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.419232    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db08cf7-f986-41e1-8a22-61080f6a7576" containerName="payload-processing"
Nov 03 20:47:01.419284 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.419251    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="85b7ad71-e155-481e-8032-8198db21ce7a" containerName="payload-processing"
Nov 03 20:47:01.425385 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.425363    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj"
Nov 03 20:47:01.434906 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.434880    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj"]
Nov 03 20:47:01.447456 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.447430    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-bf7965\""
Nov 03 20:47:01.447688 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.447532    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-bf7965-dockercfg-9v229\""
Nov 03 20:47:01.454571 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.454544    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr"]
Nov 03 20:47:01.461192 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.461155    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr"
Nov 03 20:47:01.475718 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.475694    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr"]
Nov 03 20:47:01.528507 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.528463    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a1011b5-fc48-46d6-845c-556012597a9c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj\" (UID: \"3a1011b5-fc48-46d6-845c-556012597a9c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj"
Nov 03 20:47:01.528751 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.528678    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r8k8\" (UniqueName: \"kubernetes.io/projected/3a1011b5-fc48-46d6-845c-556012597a9c-kube-api-access-7r8k8\") pod \"payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj\" (UID: \"3a1011b5-fc48-46d6-845c-556012597a9c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj"
Nov 03 20:47:01.528838 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.528804    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm866\" (UniqueName: \"kubernetes.io/projected/cf223c45-abe7-4e8c-88a3-b41ae1f1f25c-kube-api-access-vm866\") pod \"payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr\" (UID: \"cf223c45-abe7-4e8c-88a3-b41ae1f1f25c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr"
Nov 03 20:47:01.528972 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.528946    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf223c45-abe7-4e8c-88a3-b41ae1f1f25c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr\" (UID: \"cf223c45-abe7-4e8c-88a3-b41ae1f1f25c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr"
Nov 03 20:47:01.654620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.653897    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r8k8\" (UniqueName: \"kubernetes.io/projected/3a1011b5-fc48-46d6-845c-556012597a9c-kube-api-access-7r8k8\") pod \"payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj\" (UID: \"3a1011b5-fc48-46d6-845c-556012597a9c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj"
Nov 03 20:47:01.654620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.654048    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm866\" (UniqueName: \"kubernetes.io/projected/cf223c45-abe7-4e8c-88a3-b41ae1f1f25c-kube-api-access-vm866\") pod \"payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr\" (UID: \"cf223c45-abe7-4e8c-88a3-b41ae1f1f25c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr"
Nov 03 20:47:01.654620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.654176    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf223c45-abe7-4e8c-88a3-b41ae1f1f25c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr\" (UID: \"cf223c45-abe7-4e8c-88a3-b41ae1f1f25c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr"
Nov 03 20:47:01.654620 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.654359    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a1011b5-fc48-46d6-845c-556012597a9c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj\" (UID: \"3a1011b5-fc48-46d6-845c-556012597a9c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj"
Nov 03 20:47:01.686317 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.655261    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a1011b5-fc48-46d6-845c-556012597a9c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj\" (UID: \"3a1011b5-fc48-46d6-845c-556012597a9c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj"
Nov 03 20:47:01.686317 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.656443    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf223c45-abe7-4e8c-88a3-b41ae1f1f25c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr\" (UID: \"cf223c45-abe7-4e8c-88a3-b41ae1f1f25c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr"
Nov 03 20:47:01.695671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.691436    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm866\" (UniqueName: \"kubernetes.io/projected/cf223c45-abe7-4e8c-88a3-b41ae1f1f25c-kube-api-access-vm866\") pod \"payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr\" (UID: \"cf223c45-abe7-4e8c-88a3-b41ae1f1f25c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr"
Nov 03 20:47:01.715398 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.715323    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r8k8\" (UniqueName: \"kubernetes.io/projected/3a1011b5-fc48-46d6-845c-556012597a9c-kube-api-access-7r8k8\") pod \"payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj\" (UID: \"3a1011b5-fc48-46d6-845c-556012597a9c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj"
Nov 03 20:47:01.747174 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.746891    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj"
Nov 03 20:47:01.791960 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:01.791927    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr"
Nov 03 20:47:02.138469 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:47:02.138437    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1011b5_fc48_46d6_845c_556012597a9c.slice/crio-771cb841c0831c15c3606651c146cdba07cc21af09f494e7d5c38bbfd40a125d WatchSource:0}: Error finding container 771cb841c0831c15c3606651c146cdba07cc21af09f494e7d5c38bbfd40a125d: Status 404 returned error can't find the container with id 771cb841c0831c15c3606651c146cdba07cc21af09f494e7d5c38bbfd40a125d
Nov 03 20:47:02.140244 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:02.140069    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj"]
Nov 03 20:47:02.150517 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:47:02.150461    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf223c45_abe7_4e8c_88a3_b41ae1f1f25c.slice/crio-27d061510abbf56cc3bc31452c3a6c63685bd088be8e20cd7d149b090e08e0db WatchSource:0}: Error finding container 27d061510abbf56cc3bc31452c3a6c63685bd088be8e20cd7d149b090e08e0db: Status 404 returned error can't find the container with id 27d061510abbf56cc3bc31452c3a6c63685bd088be8e20cd7d149b090e08e0db
Nov 03 20:47:02.151314 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:02.151286    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr"]
Nov 03 20:47:02.475923 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:02.475879    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr" event={"ID":"cf223c45-abe7-4e8c-88a3-b41ae1f1f25c","Type":"ContainerStarted","Data":"72f134bb42a0ec9dd21e225a2b55f0f3d9045374d76980a9a8dcb1f7f5f23635"}
Nov 03 20:47:02.476455 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:02.476430    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr" event={"ID":"cf223c45-abe7-4e8c-88a3-b41ae1f1f25c","Type":"ContainerStarted","Data":"27d061510abbf56cc3bc31452c3a6c63685bd088be8e20cd7d149b090e08e0db"}
Nov 03 20:47:02.476571 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:02.476475    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr"
Nov 03 20:47:02.478670 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:02.478523    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj" event={"ID":"3a1011b5-fc48-46d6-845c-556012597a9c","Type":"ContainerStarted","Data":"3bef8f5f405d70b13447b42c8c243a3a0fad7c1a3467c7a65dd0e9d9d47dd308"}
Nov 03 20:47:02.478670 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:02.478575    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj"
Nov 03 20:47:02.478670 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:02.478638    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj" event={"ID":"3a1011b5-fc48-46d6-845c-556012597a9c","Type":"ContainerStarted","Data":"771cb841c0831c15c3606651c146cdba07cc21af09f494e7d5c38bbfd40a125d"}
Nov 03 20:47:02.508815 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:02.508756    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr" podStartSLOduration=1.508737536 podStartE2EDuration="1.508737536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:02.496697792 +0000 UTC m=+2752.629826263" watchObservedRunningTime="2026-11-03 20:47:02.508737536 +0000 UTC m=+2752.641865999"
Nov 03 20:47:02.524413 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:02.524366    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj" podStartSLOduration=1.524355103 podStartE2EDuration="1.524355103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:02.519408643 +0000 UTC m=+2752.652537101" watchObservedRunningTime="2026-11-03 20:47:02.524355103 +0000 UTC m=+2752.657483565"
Nov 03 20:47:13.490616 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:13.490573    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr"
Nov 03 20:47:13.491044 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:13.490666    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj"
Nov 03 20:47:19.633830 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:47:19.633787    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:47:32.227901 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:32.227867    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc"]
Nov 03 20:47:32.234772 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:32.234750    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc"
Nov 03 20:47:32.240177 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:32.240003    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-421604\"/\"agc-cred-0c2c8ae114be\""
Nov 03 20:47:32.241148 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:32.241121    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-421604\"/\"payload-processing-external-model-plugins-e2e-extmod-b-421604\""
Nov 03 20:47:32.241301 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:32.241176    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-421604\"/\"openshift-service-ca.crt\""
Nov 03 20:47:32.241553 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:32.241532    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-421604\"/\"routing-overlay\""
Nov 03 20:47:32.241739 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:32.241651    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-421604\"/\"kube-root-ca.crt\""
Nov 03 20:47:32.241812 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:32.241766    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-421604\"/\"payload-processing-external-model-e2e--4ce1836f-dockercfg-hzzv9\""
Nov 03 20:47:32.243579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:32.243552    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc"]
Nov 03 20:47:32.396024 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:32.395992    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k54pl\" (UniqueName: \"kubernetes.io/projected/62ea7aad-d993-4474-b5cd-b6f0239b9734-kube-api-access-k54pl\") pod \"payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc\" (UID: \"62ea7aad-d993-4474-b5cd-b6f0239b9734\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc"
Nov 03 20:47:32.396202 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:32.396041    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62ea7aad-d993-4474-b5cd-b6f0239b9734-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc\" (UID: \"62ea7aad-d993-4474-b5cd-b6f0239b9734\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc"
Nov 03 20:47:32.396202 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:32.396100    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62ea7aad-d993-4474-b5cd-b6f0239b9734-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc\" (UID: \"62ea7aad-d993-4474-b5cd-b6f0239b9734\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc"
Nov 03 20:47:32.396202 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:32.396133    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ea7aad-d993-4474-b5cd-b6f0239b9734-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc\" (UID: \"62ea7aad-d993-4474-b5cd-b6f0239b9734\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc"
Nov 03 20:47:32.498523 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:32.498445    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k54pl\" (UniqueName: \"kubernetes.io/projected/62ea7aad-d993-4474-b5cd-b6f0239b9734-kube-api-access-k54pl\") pod \"payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc\" (UID: \"62ea7aad-d993-4474-b5cd-b6f0239b9734\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc"
Nov 03 20:47:32.498523 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:32.498502    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62ea7aad-d993-4474-b5cd-b6f0239b9734-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc\" (UID: \"62ea7aad-d993-4474-b5cd-b6f0239b9734\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc"
Nov 03 20:47:32.498809 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:32.498532    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62ea7aad-d993-4474-b5cd-b6f0239b9734-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc\" (UID: \"62ea7aad-d993-4474-b5cd-b6f0239b9734\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc"
Nov 03 20:47:32.498809 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:32.498576    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ea7aad-d993-4474-b5cd-b6f0239b9734-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc\" (UID: \"62ea7aad-d993-4474-b5cd-b6f0239b9734\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc"
Nov 03 20:47:32.499384 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:32.499359    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62ea7aad-d993-4474-b5cd-b6f0239b9734-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc\" (UID: \"62ea7aad-d993-4474-b5cd-b6f0239b9734\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc"
Nov 03 20:47:32.499450 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:32.499435    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ea7aad-d993-4474-b5cd-b6f0239b9734-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc\" (UID: \"62ea7aad-d993-4474-b5cd-b6f0239b9734\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc"
Nov 03 20:47:32.501220 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:32.501198    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62ea7aad-d993-4474-b5cd-b6f0239b9734-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc\" (UID: \"62ea7aad-d993-4474-b5cd-b6f0239b9734\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc"
Nov 03 20:47:32.510782 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:32.510759    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k54pl\" (UniqueName: \"kubernetes.io/projected/62ea7aad-d993-4474-b5cd-b6f0239b9734-kube-api-access-k54pl\") pod \"payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc\" (UID: \"62ea7aad-d993-4474-b5cd-b6f0239b9734\") " pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc"
Nov 03 20:47:32.563031 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:32.563006    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc"
Nov 03 20:47:32.894327 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:47:32.894298    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ea7aad_d993_4474_b5cd_b6f0239b9734.slice/crio-a9e8609528d21d0499acd6198e3632d34094e4d311453e454843bdf943185e8f WatchSource:0}: Error finding container a9e8609528d21d0499acd6198e3632d34094e4d311453e454843bdf943185e8f: Status 404 returned error can't find the container with id a9e8609528d21d0499acd6198e3632d34094e4d311453e454843bdf943185e8f
Nov 03 20:47:32.895304 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:32.895278    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc"]
Nov 03 20:47:33.844501 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:33.844466    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc" event={"ID":"62ea7aad-d993-4474-b5cd-b6f0239b9734","Type":"ContainerStarted","Data":"5ab80eca160c855ba682ad7d8aadc6d649d172866593f6f2f1f5a35d55089d34"}
Nov 03 20:47:33.845003 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:33.844525    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc" event={"ID":"62ea7aad-d993-4474-b5cd-b6f0239b9734","Type":"ContainerStarted","Data":"a9e8609528d21d0499acd6198e3632d34094e4d311453e454843bdf943185e8f"}
Nov 03 20:47:33.845003 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:33.844640    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc"
Nov 03 20:47:33.867150 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:33.867099    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc" podStartSLOduration=1.867085637 podStartE2EDuration="1.867085637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:33.864046614 +0000 UTC m=+2783.997175077" watchObservedRunningTime="2026-11-03 20:47:33.867085637 +0000 UTC m=+2784.000214098"
Nov 03 20:47:44.855310 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:44.855279    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc"
Nov 03 20:47:49.676621 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:47:49.662224    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:47:55.096624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:47:55.083733    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc"]
Nov 03 20:48:00.084862 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:00.084818    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc" podUID="62ea7aad-d993-4474-b5cd-b6f0239b9734" containerName="payload-processing" containerID="cri-o://5ab80eca160c855ba682ad7d8aadc6d649d172866593f6f2f1f5a35d55089d34" gracePeriod=25
Nov 03 20:48:00.262884 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:00.262853    4421 generic.go:356] "Generic (PLEG): container finished" podID="62ea7aad-d993-4474-b5cd-b6f0239b9734" containerID="5ab80eca160c855ba682ad7d8aadc6d649d172866593f6f2f1f5a35d55089d34" exitCode=0
Nov 03 20:48:00.263059 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:00.262926    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc" event={"ID":"62ea7aad-d993-4474-b5cd-b6f0239b9734","Type":"ContainerDied","Data":"5ab80eca160c855ba682ad7d8aadc6d649d172866593f6f2f1f5a35d55089d34"}
Nov 03 20:48:00.392455 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:00.392423    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc"
Nov 03 20:48:00.492910 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:00.492880    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ea7aad-d993-4474-b5cd-b6f0239b9734-provider-credentials\" (UniqueName: \"kubernetes.io/projected/62ea7aad-d993-4474-b5cd-b6f0239b9734-provider-credentials\") pod \"62ea7aad-d993-4474-b5cd-b6f0239b9734\" (UID: \"62ea7aad-d993-4474-b5cd-b6f0239b9734\") "
Nov 03 20:48:00.493114 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:00.492928    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62ea7aad-d993-4474-b5cd-b6f0239b9734-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ea7aad-d993-4474-b5cd-b6f0239b9734-plugins-config-volume\") pod \"62ea7aad-d993-4474-b5cd-b6f0239b9734\" (UID: \"62ea7aad-d993-4474-b5cd-b6f0239b9734\") "
Nov 03 20:48:00.493187 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:00.493126    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ea7aad-d993-4474-b5cd-b6f0239b9734-kube-api-access-k54pl\" (UniqueName: \"kubernetes.io/projected/62ea7aad-d993-4474-b5cd-b6f0239b9734-kube-api-access-k54pl\") pod \"62ea7aad-d993-4474-b5cd-b6f0239b9734\" (UID: \"62ea7aad-d993-4474-b5cd-b6f0239b9734\") "
Nov 03 20:48:00.493187 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:00.493157    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62ea7aad-d993-4474-b5cd-b6f0239b9734-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62ea7aad-d993-4474-b5cd-b6f0239b9734-routing-overlay-volume\") pod \"62ea7aad-d993-4474-b5cd-b6f0239b9734\" (UID: \"62ea7aad-d993-4474-b5cd-b6f0239b9734\") "
Nov 03 20:48:00.493413 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:00.493376    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ea7aad-d993-4474-b5cd-b6f0239b9734-plugins-config-volume" pod "62ea7aad-d993-4474-b5cd-b6f0239b9734" (UID: "62ea7aad-d993-4474-b5cd-b6f0239b9734"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:48:00.493608 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:00.493576    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ea7aad-d993-4474-b5cd-b6f0239b9734-routing-overlay-volume" pod "62ea7aad-d993-4474-b5cd-b6f0239b9734" (UID: "62ea7aad-d993-4474-b5cd-b6f0239b9734"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:48:00.494040 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:00.494021    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ea7aad-d993-4474-b5cd-b6f0239b9734-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:00.494085 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:00.494043    4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62ea7aad-d993-4474-b5cd-b6f0239b9734-routing-overlay-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:00.495516 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:00.495491    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ea7aad-d993-4474-b5cd-b6f0239b9734-provider-credentials" pod "62ea7aad-d993-4474-b5cd-b6f0239b9734" (UID: "62ea7aad-d993-4474-b5cd-b6f0239b9734"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:48:00.495648 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:00.495514    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ea7aad-d993-4474-b5cd-b6f0239b9734-kube-api-access-k54pl" pod "62ea7aad-d993-4474-b5cd-b6f0239b9734" (UID: "62ea7aad-d993-4474-b5cd-b6f0239b9734"). InnerVolumeSpecName "kube-api-access-k54pl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:48:00.595968 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:00.595937    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k54pl\" (UniqueName: \"kubernetes.io/projected/62ea7aad-d993-4474-b5cd-b6f0239b9734-kube-api-access-k54pl\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:00.595968 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:00.595965    4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62ea7aad-d993-4474-b5cd-b6f0239b9734-provider-credentials\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:01.273208 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:01.273176    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc"
Nov 03 20:48:01.273208 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:01.273189    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc" event={"ID":"62ea7aad-d993-4474-b5cd-b6f0239b9734","Type":"ContainerDied","Data":"a9e8609528d21d0499acd6198e3632d34094e4d311453e454843bdf943185e8f"}
Nov 03 20:48:01.273765 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:01.273246    4421 scope.go:122] "RemoveContainer" containerID="5ab80eca160c855ba682ad7d8aadc6d649d172866593f6f2f1f5a35d55089d34"
Nov 03 20:48:01.301139 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:01.301113    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc"]
Nov 03 20:48:01.305538 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:01.305515    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-421604/payload-processing-external-model-e2e-extmod-b-421604-77dfhxmhc"]
Nov 03 20:48:02.018896 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:02.018863    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62ea7aad-d993-4474-b5cd-b6f0239b9734" path="/var/lib/kubelet/pods/62ea7aad-d993-4474-b5cd-b6f0239b9734/volumes"
Nov 03 20:48:07.389217 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:07.389182    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh"]
Nov 03 20:48:07.390153 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:07.390134    4421 cpu_manager.go:414] "containerMap: removing container" podUID="62ea7aad-d993-4474-b5cd-b6f0239b9734" containerName="payload-processing"
Nov 03 20:48:07.390153 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:07.390152    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ea7aad-d993-4474-b5cd-b6f0239b9734" containerName="payload-processing"
Nov 03 20:48:07.390342 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:07.390299    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ea7aad-d993-4474-b5cd-b6f0239b9734" containerName="payload-processing"
Nov 03 20:48:07.394511 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:07.394492    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh"
Nov 03 20:48:07.400592 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:07.400527    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4d63cd\""
Nov 03 20:48:07.405199 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:07.405175    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh"]
Nov 03 20:48:07.406648 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:07.406626    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4d63cd-dockercfg-5jprp\""
Nov 03 20:48:07.435679 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:07.435644    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98"]
Nov 03 20:48:07.442755 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:07.442727    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98"
Nov 03 20:48:07.449241 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:07.449191    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98"]
Nov 03 20:48:07.532123 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:07.532079    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca13e385-d5ab-41fe-87ce-7d903a01ee43-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh\" (UID: \"ca13e385-d5ab-41fe-87ce-7d903a01ee43\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh"
Nov 03 20:48:07.532333 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:07.532227    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wflg4\" (UniqueName: \"kubernetes.io/projected/6e96d27c-9c01-4e0b-b036-626ad4c4c08b-kube-api-access-wflg4\") pod \"payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98\" (UID: \"6e96d27c-9c01-4e0b-b036-626ad4c4c08b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98"
Nov 03 20:48:07.532405 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:07.532349    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e96d27c-9c01-4e0b-b036-626ad4c4c08b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98\" (UID: \"6e96d27c-9c01-4e0b-b036-626ad4c4c08b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98"
Nov 03 20:48:07.532405 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:07.532381    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkswx\" (UniqueName: \"kubernetes.io/projected/ca13e385-d5ab-41fe-87ce-7d903a01ee43-kube-api-access-pkswx\") pod \"payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh\" (UID: \"ca13e385-d5ab-41fe-87ce-7d903a01ee43\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh"
Nov 03 20:48:07.649675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:07.648853    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca13e385-d5ab-41fe-87ce-7d903a01ee43-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh\" (UID: \"ca13e385-d5ab-41fe-87ce-7d903a01ee43\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh"
Nov 03 20:48:07.649675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:07.648998    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wflg4\" (UniqueName: \"kubernetes.io/projected/6e96d27c-9c01-4e0b-b036-626ad4c4c08b-kube-api-access-wflg4\") pod \"payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98\" (UID: \"6e96d27c-9c01-4e0b-b036-626ad4c4c08b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98"
Nov 03 20:48:07.649675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:07.649115    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e96d27c-9c01-4e0b-b036-626ad4c4c08b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98\" (UID: \"6e96d27c-9c01-4e0b-b036-626ad4c4c08b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98"
Nov 03 20:48:07.649675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:07.649150    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkswx\" (UniqueName: \"kubernetes.io/projected/ca13e385-d5ab-41fe-87ce-7d903a01ee43-kube-api-access-pkswx\") pod \"payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh\" (UID: \"ca13e385-d5ab-41fe-87ce-7d903a01ee43\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh"
Nov 03 20:48:07.674625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:07.650487    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca13e385-d5ab-41fe-87ce-7d903a01ee43-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh\" (UID: \"ca13e385-d5ab-41fe-87ce-7d903a01ee43\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh"
Nov 03 20:48:07.674625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:07.651502    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e96d27c-9c01-4e0b-b036-626ad4c4c08b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98\" (UID: \"6e96d27c-9c01-4e0b-b036-626ad4c4c08b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98"
Nov 03 20:48:07.789623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:07.787937    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkswx\" (UniqueName: \"kubernetes.io/projected/ca13e385-d5ab-41fe-87ce-7d903a01ee43-kube-api-access-pkswx\") pod \"payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh\" (UID: \"ca13e385-d5ab-41fe-87ce-7d903a01ee43\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh"
Nov 03 20:48:07.806185 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:07.802813    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wflg4\" (UniqueName: \"kubernetes.io/projected/6e96d27c-9c01-4e0b-b036-626ad4c4c08b-kube-api-access-wflg4\") pod \"payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98\" (UID: \"6e96d27c-9c01-4e0b-b036-626ad4c4c08b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98"
Nov 03 20:48:08.020354 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:08.020325    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh"
Nov 03 20:48:08.072242 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:08.072212    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98"
Nov 03 20:48:08.307375 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:48:08.307344    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca13e385_d5ab_41fe_87ce_7d903a01ee43.slice/crio-3da672e750beaf92790213244120cedf8097f234150e5c005747277b7e44ce5f WatchSource:0}: Error finding container 3da672e750beaf92790213244120cedf8097f234150e5c005747277b7e44ce5f: Status 404 returned error can't find the container with id 3da672e750beaf92790213244120cedf8097f234150e5c005747277b7e44ce5f
Nov 03 20:48:08.308454 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:08.308431    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh"]
Nov 03 20:48:08.336586 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:08.336561    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh" event={"ID":"ca13e385-d5ab-41fe-87ce-7d903a01ee43","Type":"ContainerStarted","Data":"3da672e750beaf92790213244120cedf8097f234150e5c005747277b7e44ce5f"}
Nov 03 20:48:08.371915 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:48:08.371889    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e96d27c_9c01_4e0b_b036_626ad4c4c08b.slice/crio-486627c791c20895e00e924ab77c5521d3ebfb0316d53cdfa62f7fab8840f398 WatchSource:0}: Error finding container 486627c791c20895e00e924ab77c5521d3ebfb0316d53cdfa62f7fab8840f398: Status 404 returned error can't find the container with id 486627c791c20895e00e924ab77c5521d3ebfb0316d53cdfa62f7fab8840f398
Nov 03 20:48:08.372690 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:08.372668    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98"]
Nov 03 20:48:09.347826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:09.347785    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98" event={"ID":"6e96d27c-9c01-4e0b-b036-626ad4c4c08b","Type":"ContainerStarted","Data":"020d55323d343e6ca670d5d08e0bf6a179e1198de1fe835e5ec021b613d830db"}
Nov 03 20:48:09.348277 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:09.347843    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98" event={"ID":"6e96d27c-9c01-4e0b-b036-626ad4c4c08b","Type":"ContainerStarted","Data":"486627c791c20895e00e924ab77c5521d3ebfb0316d53cdfa62f7fab8840f398"}
Nov 03 20:48:09.348277 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:09.347870    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98"
Nov 03 20:48:09.349783 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:09.349759    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh" event={"ID":"ca13e385-d5ab-41fe-87ce-7d903a01ee43","Type":"ContainerStarted","Data":"df0606a40e66596682cb92fed2c0edd589465e340573db647c5d956db7c4d5cb"}
Nov 03 20:48:09.349945 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:09.349852    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh"
Nov 03 20:48:09.372142 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:09.372098    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98" podStartSLOduration=2.372086289 podStartE2EDuration="2.372086289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:09.366984527 +0000 UTC m=+2819.500112990" watchObservedRunningTime="2026-11-03 20:48:09.372086289 +0000 UTC m=+2819.505214751"
Nov 03 20:48:09.391140 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:09.391092    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh" podStartSLOduration=2.391080087 podStartE2EDuration="2.391080087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:09.387269812 +0000 UTC m=+2819.520398274" watchObservedRunningTime="2026-11-03 20:48:09.391080087 +0000 UTC m=+2819.524208548"
Nov 03 20:48:19.631985 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:48:19.631898    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:48:20.381232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:20.381201    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh"
Nov 03 20:48:20.381443 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:20.381355    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98"
Nov 03 20:48:22.021556 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.021529    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"]
Nov 03 20:48:22.027791 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.027770    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.033295 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.033268    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8c6934-openshift-default-dockercfg-jgdtj\""
Nov 03 20:48:22.038624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.038581    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"]
Nov 03 20:48:22.102335 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.102295    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-workload-certs\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.102514 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.102373    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-envoy\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.102569 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.102515    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istiod-ca-cert\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.102701 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.102681    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-credential-socket\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.102764 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.102749    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-podinfo\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.102961 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.102939    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-workload-socket\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.103019 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.103002    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-token\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.103061 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.103033    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzqn9\" (UniqueName: \"kubernetes.io/projected/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-kube-api-access-nzqn9\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.103108 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.103080    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-data\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.204822 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.204788    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-workload-certs\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.205010 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.204846    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-envoy\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.205010 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.204902    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istiod-ca-cert\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.205201 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.204999    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-credential-socket\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.205201 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.205088    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-podinfo\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.205830 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.205208    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-workload-certs\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.205830 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.205226    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-workload-socket\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.205830 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.205299    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-token\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.205830 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.205330    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzqn9\" (UniqueName: \"kubernetes.io/projected/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-kube-api-access-nzqn9\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.205830 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.205432    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-credential-socket\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.205830 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.205439    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-data\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.205830 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.205505    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-workload-socket\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.205830 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.205746    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-data\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.206164 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.205859    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istiod-ca-cert\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.207729 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.207702    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-envoy\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.208152 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.208121    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-podinfo\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.245624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.225187    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzqn9\" (UniqueName: \"kubernetes.io/projected/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-kube-api-access-nzqn9\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.248381 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.248296    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-token\") pod \"e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") " pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.358894 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.358811    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:22.651290 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:48:22.651259    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod770e86f1_0fe5_43db_b0c2_cb0c3396c8d0.slice/crio-5368875989eb972238f98952d2a938aaf4aedc40eb0aac49caa5abd02266d488 WatchSource:0}: Error finding container 5368875989eb972238f98952d2a938aaf4aedc40eb0aac49caa5abd02266d488: Status 404 returned error can't find the container with id 5368875989eb972238f98952d2a938aaf4aedc40eb0aac49caa5abd02266d488
Nov 03 20:48:22.651436 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.651288    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"]
Nov 03 20:48:22.654403 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.654373    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:48:22.654477 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.654437    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:48:22.654477 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:22.654468    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:48:23.492324 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:23.492290    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km" event={"ID":"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0","Type":"ContainerStarted","Data":"92378a3429a7319100d37bdde7df2068fe09fa82705ea1b27a06a3220f4ea11b"}
Nov 03 20:48:23.492773 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:23.492347    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km" event={"ID":"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0","Type":"ContainerStarted","Data":"5368875989eb972238f98952d2a938aaf4aedc40eb0aac49caa5abd02266d488"}
Nov 03 20:48:23.518167 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:23.518124    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km" podStartSLOduration=2.5181102859999998 podStartE2EDuration="2.518110286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:23.515164869 +0000 UTC m=+2833.648293343" watchObservedRunningTime="2026-11-03 20:48:23.518110286 +0000 UTC m=+2833.651238752"
Nov 03 20:48:24.364621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:24.363703    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:24.368852 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:24.366705    4421 patch_prober.go:29] interesting pod/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 03 20:48:24.368852 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:24.366758    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km" podUID="770e86f1-0fe5-43db-b0c2-cb0c3396c8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 03 20:48:25.361314 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:25.361282    4421 patch_prober.go:29] interesting pod/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 03 20:48:25.362088 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:25.362059    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km" podUID="770e86f1-0fe5-43db-b0c2-cb0c3396c8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 03 20:48:26.366007 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:26.365971    4421 patch_prober.go:29] interesting pod/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 03 20:48:26.366715 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:26.366031    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km" podUID="770e86f1-0fe5-43db-b0c2-cb0c3396c8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 03 20:48:27.367627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:27.364808    4421 patch_prober.go:29] interesting pod/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 03 20:48:27.367627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:27.364908    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km" podUID="770e86f1-0fe5-43db-b0c2-cb0c3396c8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 03 20:48:28.364624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:28.360072    4421 patch_prober.go:29] interesting pod/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 03 20:48:28.364624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:28.360142    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km" podUID="770e86f1-0fe5-43db-b0c2-cb0c3396c8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 03 20:48:29.367004 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:29.366969    4421 patch_prober.go:29] interesting pod/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 03 20:48:29.367680 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:29.367648    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km" podUID="770e86f1-0fe5-43db-b0c2-cb0c3396c8d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 03 20:48:30.364177 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:30.364148    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:30.364709 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:30.364686    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:30.365230 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:48:30.365213    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:48:49.630726 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:48:49.630622    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:49:03.130135 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:49:03.130089    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v"]
Nov 03 20:49:03.135938 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:49:03.135906    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v"
Nov 03 20:49:03.142093 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:49:03.142060    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-519a78\""
Nov 03 20:49:03.165726 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:49:03.165679    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v"]
Nov 03 20:49:03.260584 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:49:03.260552    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvb5x\" (UniqueName: \"kubernetes.io/projected/f399b4d8-6505-4da5-a9d2-e772cc7071d2-kube-api-access-wvb5x\") pod \"payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v\" (UID: \"f399b4d8-6505-4da5-a9d2-e772cc7071d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v"
Nov 03 20:49:03.260804 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:49:03.260629    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f399b4d8-6505-4da5-a9d2-e772cc7071d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v\" (UID: \"f399b4d8-6505-4da5-a9d2-e772cc7071d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v"
Nov 03 20:49:03.401502 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:49:03.377193    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvb5x\" (UniqueName: \"kubernetes.io/projected/f399b4d8-6505-4da5-a9d2-e772cc7071d2-kube-api-access-wvb5x\") pod \"payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v\" (UID: \"f399b4d8-6505-4da5-a9d2-e772cc7071d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v"
Nov 03 20:49:03.402550 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:49:03.402517    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f399b4d8-6505-4da5-a9d2-e772cc7071d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v\" (UID: \"f399b4d8-6505-4da5-a9d2-e772cc7071d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v"
Nov 03 20:49:03.403780 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:49:03.403756    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f399b4d8-6505-4da5-a9d2-e772cc7071d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v\" (UID: \"f399b4d8-6505-4da5-a9d2-e772cc7071d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v"
Nov 03 20:49:03.456736 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:49:03.456699    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvb5x\" (UniqueName: \"kubernetes.io/projected/f399b4d8-6505-4da5-a9d2-e772cc7071d2-kube-api-access-wvb5x\") pod \"payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v\" (UID: \"f399b4d8-6505-4da5-a9d2-e772cc7071d2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v"
Nov 03 20:49:03.469267 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:49:03.469224    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v"
Nov 03 20:49:03.887260 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:49:03.887226    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v"]
Nov 03 20:49:03.887399 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:49:03.887302    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf399b4d8_6505_4da5_a9d2_e772cc7071d2.slice/crio-b09938c920aa37c7f86afa946f82022f1de47cd79e366417dd96c03195b38875 WatchSource:0}: Error finding container b09938c920aa37c7f86afa946f82022f1de47cd79e366417dd96c03195b38875: Status 404 returned error can't find the container with id b09938c920aa37c7f86afa946f82022f1de47cd79e366417dd96c03195b38875
Nov 03 20:49:04.071506 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:49:04.071464    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v" event={"ID":"f399b4d8-6505-4da5-a9d2-e772cc7071d2","Type":"ContainerStarted","Data":"e5e2f217f943d3c4a96d1dd84ad570f9ea62e6c7aff1da7119bafd98c1b3acf5"}
Nov 03 20:49:04.071726 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:49:04.071537    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v"
Nov 03 20:49:04.071726 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:49:04.071574    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v" event={"ID":"f399b4d8-6505-4da5-a9d2-e772cc7071d2","Type":"ContainerStarted","Data":"b09938c920aa37c7f86afa946f82022f1de47cd79e366417dd96c03195b38875"}
Nov 03 20:49:04.093268 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:49:04.093217    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v" podStartSLOduration=1.093204222 podStartE2EDuration="1.093204222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:49:04.089716271 +0000 UTC m=+2874.222844727" watchObservedRunningTime="2026-11-03 20:49:04.093204222 +0000 UTC m=+2874.226332678"
Nov 03 20:49:15.083427 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:49:15.083396    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v"
Nov 03 20:49:19.663780 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:49:19.663745    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:49:49.632075 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:49:49.631981    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:50:19.630227 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:50:19.630181    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:50:49.631504 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:50:49.631469    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:50:50.220205 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:50.220174    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"]
Nov 03 20:50:50.220434 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:50.220399    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz" podUID="873d9cbd-c45e-4c14-90bb-e0ca79c56e7a" containerName="istio-proxy" containerID="cri-o://d04db38f84c76802ee960c31b7f975c6f0af370de4de81443983e2c554fea558" gracePeriod=30
Nov 03 20:50:52.625525 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:52.625495    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"]
Nov 03 20:50:52.626000 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:52.625703    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr" podUID="bac03c56-aa2e-4ba9-9841-4a9122b7fddb" containerName="istio-proxy" containerID="cri-o://6e897b58d48b0137a0ac8f1579c6ed61bfd9f2dff70e17e3780c204a4977ad6c" gracePeriod=30
Nov 03 20:50:55.554917 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.554877    4421 generic.go:356] "Generic (PLEG): container finished" podID="873d9cbd-c45e-4c14-90bb-e0ca79c56e7a" containerID="d04db38f84c76802ee960c31b7f975c6f0af370de4de81443983e2c554fea558" exitCode=0
Nov 03 20:50:55.555263 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.554920    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz" event={"ID":"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a","Type":"ContainerDied","Data":"d04db38f84c76802ee960c31b7f975c6f0af370de4de81443983e2c554fea558"}
Nov 03 20:50:55.588804 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.588773    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:50:55.714707 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.714676    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-workload-certs\") pod \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") "
Nov 03 20:50:55.714900 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.714851    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-data\") pod \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") "
Nov 03 20:50:55.714900 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.714885    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-podinfo\") pod \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") "
Nov 03 20:50:55.715041 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.714912    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-workload-socket\") pod \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") "
Nov 03 20:50:55.715041 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.714945    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-token\" (UniqueName: \"kubernetes.io/projected/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-token\") pod \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") "
Nov 03 20:50:55.715041 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.714992    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-workload-certs" pod "873d9cbd-c45e-4c14-90bb-e0ca79c56e7a" (UID: "873d9cbd-c45e-4c14-90bb-e0ca79c56e7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:50:55.715192 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.715072    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-credential-socket\") pod \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") "
Nov 03 20:50:55.715192 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.715091    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-data" pod "873d9cbd-c45e-4c14-90bb-e0ca79c56e7a" (UID: "873d9cbd-c45e-4c14-90bb-e0ca79c56e7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:50:55.715192 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.715131    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-kube-api-access-hz795\" (UniqueName: \"kubernetes.io/projected/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-kube-api-access-hz795\") pod \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") "
Nov 03 20:50:55.715192 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.715177    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-envoy\") pod \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") "
Nov 03 20:50:55.715432 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.715221    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-workload-socket" pod "873d9cbd-c45e-4c14-90bb-e0ca79c56e7a" (UID: "873d9cbd-c45e-4c14-90bb-e0ca79c56e7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:50:55.715432 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.715250    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istiod-ca-cert\") pod \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\" (UID: \"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a\") "
Nov 03 20:50:55.715432 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.715420    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-credential-socket" pod "873d9cbd-c45e-4c14-90bb-e0ca79c56e7a" (UID: "873d9cbd-c45e-4c14-90bb-e0ca79c56e7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:50:55.715708 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.715679    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istiod-ca-cert" pod "873d9cbd-c45e-4c14-90bb-e0ca79c56e7a" (UID: "873d9cbd-c45e-4c14-90bb-e0ca79c56e7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:50:55.716362 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.716333    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:55.716488 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.716366    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:55.716488 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.716383    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:55.716488 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.716397    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:55.716488 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.716411    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:55.717503 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.717473    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-kube-api-access-hz795" pod "873d9cbd-c45e-4c14-90bb-e0ca79c56e7a" (UID: "873d9cbd-c45e-4c14-90bb-e0ca79c56e7a"). InnerVolumeSpecName "kube-api-access-hz795". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:50:55.717626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.717517    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-token" pod "873d9cbd-c45e-4c14-90bb-e0ca79c56e7a" (UID: "873d9cbd-c45e-4c14-90bb-e0ca79c56e7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:50:55.717626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.717542    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-podinfo" pod "873d9cbd-c45e-4c14-90bb-e0ca79c56e7a" (UID: "873d9cbd-c45e-4c14-90bb-e0ca79c56e7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:50:55.717626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.717588    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-envoy" pod "873d9cbd-c45e-4c14-90bb-e0ca79c56e7a" (UID: "873d9cbd-c45e-4c14-90bb-e0ca79c56e7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:50:55.817692 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.817619    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz795\" (UniqueName: \"kubernetes.io/projected/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-kube-api-access-hz795\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:55.817692 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.817645    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:55.817692 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.817657    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:55.817692 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:55.817665    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:56.565431 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:56.565394    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz" event={"ID":"873d9cbd-c45e-4c14-90bb-e0ca79c56e7a","Type":"ContainerDied","Data":"322975a6794dbfea552bc5849be2d6b5afcd931f58c86a73893d4846db921715"}
Nov 03 20:50:56.565958 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:56.565497    4421 scope.go:122] "RemoveContainer" containerID="d04db38f84c76802ee960c31b7f975c6f0af370de4de81443983e2c554fea558"
Nov 03 20:50:56.565958 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:56.565413    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"
Nov 03 20:50:56.592304 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:56.592254    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"]
Nov 03 20:50:56.600253 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:56.600223    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-hjlhz"]
Nov 03 20:50:57.921450 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:57.921415    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:50:57.983815 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:57.983770    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-podinfo\") pod \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") "
Nov 03 20:50:57.984000 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:57.983914    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-workload-certs\") pod \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") "
Nov 03 20:50:57.984000 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:57.983987    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-token\" (UniqueName: \"kubernetes.io/projected/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-token\") pod \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") "
Nov 03 20:50:57.984119 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:57.984063    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-data\") pod \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") "
Nov 03 20:50:57.984169 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:57.984127    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-credential-socket\") pod \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") "
Nov 03 20:50:57.984725 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:57.984233    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-kube-api-access-psv4p\" (UniqueName: \"kubernetes.io/projected/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-kube-api-access-psv4p\") pod \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") "
Nov 03 20:50:57.984725 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:57.984267    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-envoy\") pod \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") "
Nov 03 20:50:57.984725 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:57.984293    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-workload-certs" pod "bac03c56-aa2e-4ba9-9841-4a9122b7fddb" (UID: "bac03c56-aa2e-4ba9-9841-4a9122b7fddb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:50:57.984725 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:57.984379    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istiod-ca-cert\") pod \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") "
Nov 03 20:50:57.984725 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:57.984426    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-data" pod "bac03c56-aa2e-4ba9-9841-4a9122b7fddb" (UID: "bac03c56-aa2e-4ba9-9841-4a9122b7fddb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:50:57.984725 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:57.984457    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-workload-socket\") pod \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\" (UID: \"bac03c56-aa2e-4ba9-9841-4a9122b7fddb\") "
Nov 03 20:50:57.984725 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:57.984535    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-credential-socket" pod "bac03c56-aa2e-4ba9-9841-4a9122b7fddb" (UID: "bac03c56-aa2e-4ba9-9841-4a9122b7fddb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:50:57.985423 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:57.984903    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istiod-ca-cert" pod "bac03c56-aa2e-4ba9-9841-4a9122b7fddb" (UID: "bac03c56-aa2e-4ba9-9841-4a9122b7fddb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:50:57.985423 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:57.985145    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-workload-socket" pod "bac03c56-aa2e-4ba9-9841-4a9122b7fddb" (UID: "bac03c56-aa2e-4ba9-9841-4a9122b7fddb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:50:57.985553 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:57.985476    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:57.985553 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:57.985498    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:57.985553 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:57.985512    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:57.985553 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:57.985527    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:57.985553 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:57.985539    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:57.986893 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:57.986868    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-podinfo" pod "bac03c56-aa2e-4ba9-9841-4a9122b7fddb" (UID: "bac03c56-aa2e-4ba9-9841-4a9122b7fddb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:50:57.987261 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:57.987197    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-token" pod "bac03c56-aa2e-4ba9-9841-4a9122b7fddb" (UID: "bac03c56-aa2e-4ba9-9841-4a9122b7fddb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:50:57.987644 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:57.987580    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-kube-api-access-psv4p" pod "bac03c56-aa2e-4ba9-9841-4a9122b7fddb" (UID: "bac03c56-aa2e-4ba9-9841-4a9122b7fddb"). InnerVolumeSpecName "kube-api-access-psv4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:50:57.988722 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:57.988698    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-envoy" pod "bac03c56-aa2e-4ba9-9841-4a9122b7fddb" (UID: "bac03c56-aa2e-4ba9-9841-4a9122b7fddb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:50:58.020691 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.020657    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="873d9cbd-c45e-4c14-90bb-e0ca79c56e7a" path="/var/lib/kubelet/pods/873d9cbd-c45e-4c14-90bb-e0ca79c56e7a/volumes"
Nov 03 20:50:58.087803 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.087766    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:58.087803 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.087804    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:58.088123 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.087821    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psv4p\" (UniqueName: \"kubernetes.io/projected/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-kube-api-access-psv4p\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:58.088123 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.087837    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac03c56-aa2e-4ba9-9841-4a9122b7fddb-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:58.306195 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.306106    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45"]
Nov 03 20:50:58.306404 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.306353    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45" podUID="d3b9318c-440f-41d3-8956-837e2ca7ae62" containerName="maas-api" containerID="cri-o://53b78541282c1c218146b4b3ab4ee1784b788e2e11b112a77f671b146598a18e" gracePeriod=30
Nov 03 20:50:58.760283 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.760237    4421 generic.go:356] "Generic (PLEG): container finished" podID="d3b9318c-440f-41d3-8956-837e2ca7ae62" containerID="53b78541282c1c218146b4b3ab4ee1784b788e2e11b112a77f671b146598a18e" exitCode=0
Nov 03 20:50:58.760472 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.760378    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45" event={"ID":"d3b9318c-440f-41d3-8956-837e2ca7ae62","Type":"ContainerDied","Data":"53b78541282c1c218146b4b3ab4ee1784b788e2e11b112a77f671b146598a18e"}
Nov 03 20:50:58.760472 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.760449    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45" event={"ID":"d3b9318c-440f-41d3-8956-837e2ca7ae62","Type":"ContainerDied","Data":"4269661eef549e163e1fbd42567419f4efb984d245a4cca993fe82c524bb1f9e"}
Nov 03 20:50:58.760472 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.760470    4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4269661eef549e163e1fbd42567419f4efb984d245a4cca993fe82c524bb1f9e"
Nov 03 20:50:58.763634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.763000    4421 generic.go:356] "Generic (PLEG): container finished" podID="bac03c56-aa2e-4ba9-9841-4a9122b7fddb" containerID="6e897b58d48b0137a0ac8f1579c6ed61bfd9f2dff70e17e3780c204a4977ad6c" exitCode=0
Nov 03 20:50:58.763634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.763368    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr" event={"ID":"bac03c56-aa2e-4ba9-9841-4a9122b7fddb","Type":"ContainerDied","Data":"6e897b58d48b0137a0ac8f1579c6ed61bfd9f2dff70e17e3780c204a4977ad6c"}
Nov 03 20:50:58.763634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.763419    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr" event={"ID":"bac03c56-aa2e-4ba9-9841-4a9122b7fddb","Type":"ContainerDied","Data":"05d1014172d9401bee50e68878a3f97933b9effbadf4c648265217a99d2a7363"}
Nov 03 20:50:58.763634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.763471    4421 scope.go:122] "RemoveContainer" containerID="6e897b58d48b0137a0ac8f1579c6ed61bfd9f2dff70e17e3780c204a4977ad6c"
Nov 03 20:50:58.764149 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.763763    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"
Nov 03 20:50:58.823230 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.823199    4421 scope.go:122] "RemoveContainer" containerID="6e897b58d48b0137a0ac8f1579c6ed61bfd9f2dff70e17e3780c204a4977ad6c"
Nov 03 20:50:58.824475 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.824446    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"]
Nov 03 20:50:58.832831 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:50:58.832799    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e897b58d48b0137a0ac8f1579c6ed61bfd9f2dff70e17e3780c204a4977ad6c\": container with ID starting with 6e897b58d48b0137a0ac8f1579c6ed61bfd9f2dff70e17e3780c204a4977ad6c not found: ID does not exist" containerID="6e897b58d48b0137a0ac8f1579c6ed61bfd9f2dff70e17e3780c204a4977ad6c"
Nov 03 20:50:58.833817 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.832839    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e897b58d48b0137a0ac8f1579c6ed61bfd9f2dff70e17e3780c204a4977ad6c"} err="failed to get container status \"6e897b58d48b0137a0ac8f1579c6ed61bfd9f2dff70e17e3780c204a4977ad6c\": rpc error: code = NotFound desc = could not find container \"6e897b58d48b0137a0ac8f1579c6ed61bfd9f2dff70e17e3780c204a4977ad6c\": container with ID starting with 6e897b58d48b0137a0ac8f1579c6ed61bfd9f2dff70e17e3780c204a4977ad6c not found: ID does not exist"
Nov 03 20:50:58.845342 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.843796    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45"
Nov 03 20:50:58.854766 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.854740    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-lv9vr"]
Nov 03 20:50:58.947629 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.947583    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3b9318c-440f-41d3-8956-837e2ca7ae62-kube-api-access-gwfrv\" (UniqueName: \"kubernetes.io/projected/d3b9318c-440f-41d3-8956-837e2ca7ae62-kube-api-access-gwfrv\") pod \"d3b9318c-440f-41d3-8956-837e2ca7ae62\" (UID: \"d3b9318c-440f-41d3-8956-837e2ca7ae62\") "
Nov 03 20:50:58.948144 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.947742    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3b9318c-440f-41d3-8956-837e2ca7ae62-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3b9318c-440f-41d3-8956-837e2ca7ae62-metrics-certs\") pod \"d3b9318c-440f-41d3-8956-837e2ca7ae62\" (UID: \"d3b9318c-440f-41d3-8956-837e2ca7ae62\") "
Nov 03 20:50:58.948144 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.947827    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3b9318c-440f-41d3-8956-837e2ca7ae62-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3b9318c-440f-41d3-8956-837e2ca7ae62-maas-api-tls\") pod \"d3b9318c-440f-41d3-8956-837e2ca7ae62\" (UID: \"d3b9318c-440f-41d3-8956-837e2ca7ae62\") "
Nov 03 20:50:58.948144 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.947861    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3b9318c-440f-41d3-8956-837e2ca7ae62-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3b9318c-440f-41d3-8956-837e2ca7ae62-openshift-service-ca\") pod \"d3b9318c-440f-41d3-8956-837e2ca7ae62\" (UID: \"d3b9318c-440f-41d3-8956-837e2ca7ae62\") "
Nov 03 20:50:58.948749 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.948667    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3b9318c-440f-41d3-8956-837e2ca7ae62-openshift-service-ca" pod "d3b9318c-440f-41d3-8956-837e2ca7ae62" (UID: "d3b9318c-440f-41d3-8956-837e2ca7ae62"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:50:58.948862 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.948839    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3b9318c-440f-41d3-8956-837e2ca7ae62-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:58.950382 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.950354    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b9318c-440f-41d3-8956-837e2ca7ae62-kube-api-access-gwfrv" pod "d3b9318c-440f-41d3-8956-837e2ca7ae62" (UID: "d3b9318c-440f-41d3-8956-837e2ca7ae62"). InnerVolumeSpecName "kube-api-access-gwfrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:50:58.950501 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.950470    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3b9318c-440f-41d3-8956-837e2ca7ae62-maas-api-tls" pod "d3b9318c-440f-41d3-8956-837e2ca7ae62" (UID: "d3b9318c-440f-41d3-8956-837e2ca7ae62"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:50:58.950653 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:58.950590    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3b9318c-440f-41d3-8956-837e2ca7ae62-metrics-certs" pod "d3b9318c-440f-41d3-8956-837e2ca7ae62" (UID: "d3b9318c-440f-41d3-8956-837e2ca7ae62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:50:59.050432 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:59.050393    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwfrv\" (UniqueName: \"kubernetes.io/projected/d3b9318c-440f-41d3-8956-837e2ca7ae62-kube-api-access-gwfrv\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:59.050432 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:59.050430    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3b9318c-440f-41d3-8956-837e2ca7ae62-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:59.050688 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:59.050445    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3b9318c-440f-41d3-8956-837e2ca7ae62-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:59.773256 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:59.773231    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45"
Nov 03 20:50:59.803789 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:59.803756    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45"]
Nov 03 20:50:59.809568 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:50:59.809544    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3d4fe2-9ff59bc5d-b8m45"]
Nov 03 20:51:00.019078 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:00.019047    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bac03c56-aa2e-4ba9-9841-4a9122b7fddb" path="/var/lib/kubelet/pods/bac03c56-aa2e-4ba9-9841-4a9122b7fddb/volumes"
Nov 03 20:51:00.019526 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:00.019511    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3b9318c-440f-41d3-8956-837e2ca7ae62" path="/var/lib/kubelet/pods/d3b9318c-440f-41d3-8956-837e2ca7ae62/volumes"
Nov 03 20:51:02.961621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:02.961049    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"]
Nov 03 20:51:02.965873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:02.962750    4421 cpu_manager.go:414] "containerMap: removing container" podUID="bac03c56-aa2e-4ba9-9841-4a9122b7fddb" containerName="istio-proxy"
Nov 03 20:51:02.965873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:02.962774    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac03c56-aa2e-4ba9-9841-4a9122b7fddb" containerName="istio-proxy"
Nov 03 20:51:02.965873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:02.962809    4421 cpu_manager.go:414] "containerMap: removing container" podUID="d3b9318c-440f-41d3-8956-837e2ca7ae62" containerName="maas-api"
Nov 03 20:51:02.965873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:02.962817    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3b9318c-440f-41d3-8956-837e2ca7ae62" containerName="maas-api"
Nov 03 20:51:02.965873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:02.962888    4421 cpu_manager.go:414] "containerMap: removing container" podUID="873d9cbd-c45e-4c14-90bb-e0ca79c56e7a" containerName="istio-proxy"
Nov 03 20:51:02.965873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:02.962896    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="873d9cbd-c45e-4c14-90bb-e0ca79c56e7a" containerName="istio-proxy"
Nov 03 20:51:02.965873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:02.963099    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac03c56-aa2e-4ba9-9841-4a9122b7fddb" containerName="istio-proxy"
Nov 03 20:51:02.965873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:02.963118    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3b9318c-440f-41d3-8956-837e2ca7ae62" containerName="maas-api"
Nov 03 20:51:02.965873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:02.963136    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="873d9cbd-c45e-4c14-90bb-e0ca79c56e7a" containerName="istio-proxy"
Nov 03 20:51:02.973860 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:02.973832    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:02.976133 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:02.976114    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"]
Nov 03 20:51:02.978190 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:02.978166    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4d63cd-openshift-default-dockercfg-2j946\""
Nov 03 20:51:03.149999 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.149957    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-credential-socket\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.150195 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.150169    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-token\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.150279 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.150211    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-podinfo\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.150279 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.150242    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-workload-socket\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.150400 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.150332    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-data\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.150517 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.150491    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-workload-certs\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.150682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.150669    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-envoy\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.150755 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.150692    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153d1bbf-3857-43c6-b66e-6dca99b48073-istiod-ca-cert\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.150804 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.150747    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkxfd\" (UniqueName: \"kubernetes.io/projected/153d1bbf-3857-43c6-b66e-6dca99b48073-kube-api-access-wkxfd\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.252529 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.252444    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-envoy\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.252529 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.252486    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153d1bbf-3857-43c6-b66e-6dca99b48073-istiod-ca-cert\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.252529 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.252521    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkxfd\" (UniqueName: \"kubernetes.io/projected/153d1bbf-3857-43c6-b66e-6dca99b48073-kube-api-access-wkxfd\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.252845 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.252584    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-credential-socket\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.252845 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.252768    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-token\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.252845 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.252811    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-podinfo\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.252993 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.252867    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-workload-socket\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.252993 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.252972    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-credential-socket\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.253233 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.253208    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-workload-socket\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.253320 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.253254    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153d1bbf-3857-43c6-b66e-6dca99b48073-istiod-ca-cert\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.253320 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.253259    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-data\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.253320 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.253273    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-data\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.253488 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.253385    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-workload-certs\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.253703 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.253687    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-workload-certs\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.255103 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.255080    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-envoy\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.255529 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.255505    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-podinfo\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.265131 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.265108    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-token\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.265263 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.265242    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkxfd\" (UniqueName: \"kubernetes.io/projected/153d1bbf-3857-43c6-b66e-6dca99b48073-kube-api-access-wkxfd\") pod \"e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") " pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.291360 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.291336    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:03.585147 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:51:03.585119    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153d1bbf_3857_43c6_b66e_6dca99b48073.slice/crio-0c5c775bfa2ed98ca5313ad7c9021b92c6ea597bb3ca6e125cdc237075cef26e WatchSource:0}: Error finding container 0c5c775bfa2ed98ca5313ad7c9021b92c6ea597bb3ca6e125cdc237075cef26e: Status 404 returned error can't find the container with id 0c5c775bfa2ed98ca5313ad7c9021b92c6ea597bb3ca6e125cdc237075cef26e
Nov 03 20:51:03.586257 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.586232    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"]
Nov 03 20:51:03.588702 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.588666    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:51:03.588770 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.588753    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:51:03.588815 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.588798    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:51:03.828771 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.828729    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x" event={"ID":"153d1bbf-3857-43c6-b66e-6dca99b48073","Type":"ContainerStarted","Data":"1b883b2063c3ada7a3c27f955ef31bd585a60455af4902149a8dff9b9bae067c"}
Nov 03 20:51:03.828948 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.828790    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x" event={"ID":"153d1bbf-3857-43c6-b66e-6dca99b48073","Type":"ContainerStarted","Data":"0c5c775bfa2ed98ca5313ad7c9021b92c6ea597bb3ca6e125cdc237075cef26e"}
Nov 03 20:51:03.854132 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:03.854036    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x" podStartSLOduration=1.854021411 podStartE2EDuration="1.854021411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:03.85378415 +0000 UTC m=+2993.986912622" watchObservedRunningTime="2026-11-03 20:51:03.854021411 +0000 UTC m=+2993.987149872"
Nov 03 20:51:04.291653 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:04.291592    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:04.293208 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:04.293178    4421 patch_prober.go:29] interesting pod/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Nov 03 20:51:04.293350 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:04.293239    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x" podUID="153d1bbf-3857-43c6-b66e-6dca99b48073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Nov 03 20:51:05.307622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:05.297924    4421 patch_prober.go:29] interesting pod/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Nov 03 20:51:05.307622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:05.297997    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x" podUID="153d1bbf-3857-43c6-b66e-6dca99b48073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Nov 03 20:51:06.291843 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:06.291806    4421 patch_prober.go:29] interesting pod/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Nov 03 20:51:06.292037 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:06.291870    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x" podUID="153d1bbf-3857-43c6-b66e-6dca99b48073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Nov 03 20:51:07.292490 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:07.292453    4421 patch_prober.go:29] interesting pod/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Nov 03 20:51:07.292923 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:07.292513    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x" podUID="153d1bbf-3857-43c6-b66e-6dca99b48073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Nov 03 20:51:08.292129 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:08.292091    4421 patch_prober.go:29] interesting pod/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Nov 03 20:51:08.292324 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:08.292158    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x" podUID="153d1bbf-3857-43c6-b66e-6dca99b48073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Nov 03 20:51:09.299992 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:09.299953    4421 patch_prober.go:29] interesting pod/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Nov 03 20:51:09.300879 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:09.300558    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x" podUID="153d1bbf-3857-43c6-b66e-6dca99b48073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Nov 03 20:51:10.291860 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:10.291825    4421 patch_prober.go:29] interesting pod/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Nov 03 20:51:10.292069 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:10.291879    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x" podUID="153d1bbf-3857-43c6-b66e-6dca99b48073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Nov 03 20:51:11.275811 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:11.275777    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv"]
Nov 03 20:51:11.300035 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:11.300005    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:11.302110 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:11.302048    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:11.306036 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:11.306015    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:51:12.957252 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:12.957221    4421 scope.go:122] "RemoveContainer" containerID="53b78541282c1c218146b4b3ab4ee1784b788e2e11b112a77f671b146598a18e"
Nov 03 20:51:16.276867 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:16.276834    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv" podUID="78dabb1d-ff1a-4de3-a082-cea203b39bea" containerName="payload-processing" containerID="cri-o://3426b4f9c962d697e7ca0852ec56221a83508f062b02349afdf137586ac860ce" gracePeriod=25
Nov 03 20:51:16.591278 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:16.591247    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv"
Nov 03 20:51:16.707506 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:16.707455    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78dabb1d-ff1a-4de3-a082-cea203b39bea-provider-credentials\" (UniqueName: \"kubernetes.io/projected/78dabb1d-ff1a-4de3-a082-cea203b39bea-provider-credentials\") pod \"78dabb1d-ff1a-4de3-a082-cea203b39bea\" (UID: \"78dabb1d-ff1a-4de3-a082-cea203b39bea\") "
Nov 03 20:51:16.707506 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:16.707506    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78dabb1d-ff1a-4de3-a082-cea203b39bea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78dabb1d-ff1a-4de3-a082-cea203b39bea-plugins-config-volume\") pod \"78dabb1d-ff1a-4de3-a082-cea203b39bea\" (UID: \"78dabb1d-ff1a-4de3-a082-cea203b39bea\") "
Nov 03 20:51:16.707828 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:16.707544    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78dabb1d-ff1a-4de3-a082-cea203b39bea-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/78dabb1d-ff1a-4de3-a082-cea203b39bea-routing-overlay-volume\") pod \"78dabb1d-ff1a-4de3-a082-cea203b39bea\" (UID: \"78dabb1d-ff1a-4de3-a082-cea203b39bea\") "
Nov 03 20:51:16.707885 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:16.707833    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78dabb1d-ff1a-4de3-a082-cea203b39bea-kube-api-access-srtr5\" (UniqueName: \"kubernetes.io/projected/78dabb1d-ff1a-4de3-a082-cea203b39bea-kube-api-access-srtr5\") pod \"78dabb1d-ff1a-4de3-a082-cea203b39bea\" (UID: \"78dabb1d-ff1a-4de3-a082-cea203b39bea\") "
Nov 03 20:51:16.708000 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:16.707967    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78dabb1d-ff1a-4de3-a082-cea203b39bea-routing-overlay-volume" pod "78dabb1d-ff1a-4de3-a082-cea203b39bea" (UID: "78dabb1d-ff1a-4de3-a082-cea203b39bea"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:51:16.708000 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:16.707988    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78dabb1d-ff1a-4de3-a082-cea203b39bea-plugins-config-volume" pod "78dabb1d-ff1a-4de3-a082-cea203b39bea" (UID: "78dabb1d-ff1a-4de3-a082-cea203b39bea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:51:16.708428 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:16.708400    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78dabb1d-ff1a-4de3-a082-cea203b39bea-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:16.708428 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:16.708420    4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/78dabb1d-ff1a-4de3-a082-cea203b39bea-routing-overlay-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:16.710020 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:16.709997    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78dabb1d-ff1a-4de3-a082-cea203b39bea-kube-api-access-srtr5" pod "78dabb1d-ff1a-4de3-a082-cea203b39bea" (UID: "78dabb1d-ff1a-4de3-a082-cea203b39bea"). InnerVolumeSpecName "kube-api-access-srtr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:51:16.710148 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:16.710129    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78dabb1d-ff1a-4de3-a082-cea203b39bea-provider-credentials" pod "78dabb1d-ff1a-4de3-a082-cea203b39bea" (UID: "78dabb1d-ff1a-4de3-a082-cea203b39bea"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:51:16.811321 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:16.810978    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srtr5\" (UniqueName: \"kubernetes.io/projected/78dabb1d-ff1a-4de3-a082-cea203b39bea-kube-api-access-srtr5\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:16.811552 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:16.811531    4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/78dabb1d-ff1a-4de3-a082-cea203b39bea-provider-credentials\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:17.278249 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:17.278212    4421 generic.go:356] "Generic (PLEG): container finished" podID="78dabb1d-ff1a-4de3-a082-cea203b39bea" containerID="3426b4f9c962d697e7ca0852ec56221a83508f062b02349afdf137586ac860ce" exitCode=0
Nov 03 20:51:17.278782 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:17.278257    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv" event={"ID":"78dabb1d-ff1a-4de3-a082-cea203b39bea","Type":"ContainerDied","Data":"3426b4f9c962d697e7ca0852ec56221a83508f062b02349afdf137586ac860ce"}
Nov 03 20:51:17.278782 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:17.278324    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv" event={"ID":"78dabb1d-ff1a-4de3-a082-cea203b39bea","Type":"ContainerDied","Data":"f680daf117a4a03464f67fbec1b89ccaa53a49eaa228fb92ce5a6ba8b725b484"}
Nov 03 20:51:17.278782 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:17.278344    4421 scope.go:122] "RemoveContainer" containerID="3426b4f9c962d697e7ca0852ec56221a83508f062b02349afdf137586ac860ce"
Nov 03 20:51:17.278782 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:17.278275    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv"
Nov 03 20:51:17.292714 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:17.292681    4421 scope.go:122] "RemoveContainer" containerID="3426b4f9c962d697e7ca0852ec56221a83508f062b02349afdf137586ac860ce"
Nov 03 20:51:17.293060 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:51:17.293035    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3426b4f9c962d697e7ca0852ec56221a83508f062b02349afdf137586ac860ce\": container with ID starting with 3426b4f9c962d697e7ca0852ec56221a83508f062b02349afdf137586ac860ce not found: ID does not exist" containerID="3426b4f9c962d697e7ca0852ec56221a83508f062b02349afdf137586ac860ce"
Nov 03 20:51:17.293125 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:17.293072    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3426b4f9c962d697e7ca0852ec56221a83508f062b02349afdf137586ac860ce"} err="failed to get container status \"3426b4f9c962d697e7ca0852ec56221a83508f062b02349afdf137586ac860ce\": rpc error: code = NotFound desc = could not find container \"3426b4f9c962d697e7ca0852ec56221a83508f062b02349afdf137586ac860ce\": container with ID starting with 3426b4f9c962d697e7ca0852ec56221a83508f062b02349afdf137586ac860ce not found: ID does not exist"
Nov 03 20:51:17.344527 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:17.344496    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv"]
Nov 03 20:51:17.354576 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:17.354550    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-05473b/payload-processing-external-model-e2e-extmod-b-05473b-688b82lbv"]
Nov 03 20:51:18.018538 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:18.018508    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78dabb1d-ff1a-4de3-a082-cea203b39bea" path="/var/lib/kubelet/pods/78dabb1d-ff1a-4de3-a082-cea203b39bea/volumes"
Nov 03 20:51:19.632364 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:51:19.632281    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:51:24.064070 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:24.064038    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm"]
Nov 03 20:51:24.064996 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:24.064979    4421 cpu_manager.go:414] "containerMap: removing container" podUID="78dabb1d-ff1a-4de3-a082-cea203b39bea" containerName="payload-processing"
Nov 03 20:51:24.064996 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:24.064996    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78dabb1d-ff1a-4de3-a082-cea203b39bea" containerName="payload-processing"
Nov 03 20:51:24.065123 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:24.065114    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="78dabb1d-ff1a-4de3-a082-cea203b39bea" containerName="payload-processing"
Nov 03 20:51:24.070584 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:24.070562    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm"
Nov 03 20:51:24.076051 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:24.076029    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-745c52-dockercfg-pzspk\""
Nov 03 20:51:24.076196 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:24.076136    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-745c52\""
Nov 03 20:51:24.077497 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:24.077476    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm"]
Nov 03 20:51:24.099891 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:24.099859    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj"]
Nov 03 20:51:24.105260 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:24.105225    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj"
Nov 03 20:51:24.115919 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:24.115888    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj"]
Nov 03 20:51:24.252389 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:24.252127    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fcd696c-4d3c-484f-84e7-c6b9b7f05005-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm\" (UID: \"2fcd696c-4d3c-484f-84e7-c6b9b7f05005\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm"
Nov 03 20:51:24.252389 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:24.252373    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnz5k\" (UniqueName: \"kubernetes.io/projected/2fcd696c-4d3c-484f-84e7-c6b9b7f05005-kube-api-access-hnz5k\") pod \"payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm\" (UID: \"2fcd696c-4d3c-484f-84e7-c6b9b7f05005\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm"
Nov 03 20:51:24.252686 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:24.252457    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsplv\" (UniqueName: \"kubernetes.io/projected/d4cc153a-39be-4e4f-bd4e-0fa01599cc56-kube-api-access-wsplv\") pod \"payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj\" (UID: \"d4cc153a-39be-4e4f-bd4e-0fa01599cc56\") " pod="openshift-ingress/payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj"
Nov 03 20:51:24.252686 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:24.252541    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cc153a-39be-4e4f-bd4e-0fa01599cc56-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj\" (UID: \"d4cc153a-39be-4e4f-bd4e-0fa01599cc56\") " pod="openshift-ingress/payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj"
Nov 03 20:51:24.363019 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:24.362917    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnz5k\" (UniqueName: \"kubernetes.io/projected/2fcd696c-4d3c-484f-84e7-c6b9b7f05005-kube-api-access-hnz5k\") pod \"payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm\" (UID: \"2fcd696c-4d3c-484f-84e7-c6b9b7f05005\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm"
Nov 03 20:51:24.363279 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:24.363029    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsplv\" (UniqueName: \"kubernetes.io/projected/d4cc153a-39be-4e4f-bd4e-0fa01599cc56-kube-api-access-wsplv\") pod \"payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj\" (UID: \"d4cc153a-39be-4e4f-bd4e-0fa01599cc56\") " pod="openshift-ingress/payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj"
Nov 03 20:51:24.363279 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:24.363110    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cc153a-39be-4e4f-bd4e-0fa01599cc56-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj\" (UID: \"d4cc153a-39be-4e4f-bd4e-0fa01599cc56\") " pod="openshift-ingress/payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj"
Nov 03 20:51:24.363279 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:24.363262    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fcd696c-4d3c-484f-84e7-c6b9b7f05005-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm\" (UID: \"2fcd696c-4d3c-484f-84e7-c6b9b7f05005\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm"
Nov 03 20:51:24.364398 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:24.364293    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fcd696c-4d3c-484f-84e7-c6b9b7f05005-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm\" (UID: \"2fcd696c-4d3c-484f-84e7-c6b9b7f05005\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm"
Nov 03 20:51:24.388967 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:24.387249    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cc153a-39be-4e4f-bd4e-0fa01599cc56-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj\" (UID: \"d4cc153a-39be-4e4f-bd4e-0fa01599cc56\") " pod="openshift-ingress/payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj"
Nov 03 20:51:24.414930 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:24.414892    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnz5k\" (UniqueName: \"kubernetes.io/projected/2fcd696c-4d3c-484f-84e7-c6b9b7f05005-kube-api-access-hnz5k\") pod \"payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm\" (UID: \"2fcd696c-4d3c-484f-84e7-c6b9b7f05005\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm"
Nov 03 20:51:24.436673 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:24.436560    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsplv\" (UniqueName: \"kubernetes.io/projected/d4cc153a-39be-4e4f-bd4e-0fa01599cc56-kube-api-access-wsplv\") pod \"payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj\" (UID: \"d4cc153a-39be-4e4f-bd4e-0fa01599cc56\") " pod="openshift-ingress/payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj"
Nov 03 20:51:24.688714 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:24.688692    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm"
Nov 03 20:51:24.727379 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:24.727344    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj"
Nov 03 20:51:24.980853 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:24.980766    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm"]
Nov 03 20:51:24.985441 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:51:24.985405    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fcd696c_4d3c_484f_84e7_c6b9b7f05005.slice/crio-1928892efe71a960f5a5ce9f3ba354b940732df776f0f3857b356310f0b45c3a WatchSource:0}: Error finding container 1928892efe71a960f5a5ce9f3ba354b940732df776f0f3857b356310f0b45c3a: Status 404 returned error can't find the container with id 1928892efe71a960f5a5ce9f3ba354b940732df776f0f3857b356310f0b45c3a
Nov 03 20:51:25.063852 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:51:25.063823    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4cc153a_39be_4e4f_bd4e_0fa01599cc56.slice/crio-c6f313522ddff4a5724b3215f2c15b7713fd270f0b60e41c34f72db62bc49663 WatchSource:0}: Error finding container c6f313522ddff4a5724b3215f2c15b7713fd270f0b60e41c34f72db62bc49663: Status 404 returned error can't find the container with id c6f313522ddff4a5724b3215f2c15b7713fd270f0b60e41c34f72db62bc49663
Nov 03 20:51:25.064700 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:25.064674    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj"]
Nov 03 20:51:25.472615 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:25.472571    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm" event={"ID":"2fcd696c-4d3c-484f-84e7-c6b9b7f05005","Type":"ContainerStarted","Data":"e237a79e683ee5d47236d079930759e4e611b12649cdb8b31968879e88c59172"}
Nov 03 20:51:25.472808 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:25.472667    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm" event={"ID":"2fcd696c-4d3c-484f-84e7-c6b9b7f05005","Type":"ContainerStarted","Data":"1928892efe71a960f5a5ce9f3ba354b940732df776f0f3857b356310f0b45c3a"}
Nov 03 20:51:25.472808 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:25.472698    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm"
Nov 03 20:51:25.474770 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:25.474746    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj" event={"ID":"d4cc153a-39be-4e4f-bd4e-0fa01599cc56","Type":"ContainerStarted","Data":"bde111a976467874c07408ca2255b1db946ceb6de3f8f82592f39a798131333b"}
Nov 03 20:51:25.474900 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:25.474783    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj" event={"ID":"d4cc153a-39be-4e4f-bd4e-0fa01599cc56","Type":"ContainerStarted","Data":"c6f313522ddff4a5724b3215f2c15b7713fd270f0b60e41c34f72db62bc49663"}
Nov 03 20:51:25.474900 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:25.474809    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj"
Nov 03 20:51:25.500447 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:25.500385    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm" podStartSLOduration=1.500367942 podStartE2EDuration="1.500367942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:25.496722096 +0000 UTC m=+3015.629850559" watchObservedRunningTime="2026-11-03 20:51:25.500367942 +0000 UTC m=+3015.633496405"
Nov 03 20:51:25.531676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:25.529306    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj" podStartSLOduration=1.529287533 podStartE2EDuration="1.529287533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:25.528887434 +0000 UTC m=+3015.662015903" watchObservedRunningTime="2026-11-03 20:51:25.529287533 +0000 UTC m=+3015.662416006"
Nov 03 20:51:34.578348 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:34.578306    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q"]
Nov 03 20:51:36.487302 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:36.487275    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj"
Nov 03 20:51:36.487769 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:36.487659    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm"
Nov 03 20:51:38.213922 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:38.213878    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf"]
Nov 03 20:51:39.578707 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:39.578645    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q" podUID="4582fd72-a3a0-4d41-a3a4-29cf500022f4" containerName="payload-processing" containerID="cri-o://e0172df076c1736fe2d1ec4d682fa9a9e92ed94aefea8513f9afabba321853d5" gracePeriod=25
Nov 03 20:51:42.418652 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:42.418618    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk"]
Nov 03 20:51:42.423571 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:42.423530    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk" podUID="53370acb-4269-4c64-947d-c8f98d5beb9c" containerName="maas-api" containerID="cri-o://8470cff6cb70704156ab8ad826db61c0f11f9ea439068086397c8e0ad7bd74f5" gracePeriod=30
Nov 03 20:51:43.087804 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:43.087774    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk"
Nov 03 20:51:43.146921 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:43.146885    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53370acb-4269-4c64-947d-c8f98d5beb9c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/53370acb-4269-4c64-947d-c8f98d5beb9c-metrics-certs\") pod \"53370acb-4269-4c64-947d-c8f98d5beb9c\" (UID: \"53370acb-4269-4c64-947d-c8f98d5beb9c\") "
Nov 03 20:51:43.147233 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:43.147212    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53370acb-4269-4c64-947d-c8f98d5beb9c-kube-api-access-gmdp9\" (UniqueName: \"kubernetes.io/projected/53370acb-4269-4c64-947d-c8f98d5beb9c-kube-api-access-gmdp9\") pod \"53370acb-4269-4c64-947d-c8f98d5beb9c\" (UID: \"53370acb-4269-4c64-947d-c8f98d5beb9c\") "
Nov 03 20:51:43.147392 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:43.147248    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53370acb-4269-4c64-947d-c8f98d5beb9c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53370acb-4269-4c64-947d-c8f98d5beb9c-maas-api-tls\") pod \"53370acb-4269-4c64-947d-c8f98d5beb9c\" (UID: \"53370acb-4269-4c64-947d-c8f98d5beb9c\") "
Nov 03 20:51:43.147392 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:43.147321    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53370acb-4269-4c64-947d-c8f98d5beb9c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53370acb-4269-4c64-947d-c8f98d5beb9c-openshift-service-ca\") pod \"53370acb-4269-4c64-947d-c8f98d5beb9c\" (UID: \"53370acb-4269-4c64-947d-c8f98d5beb9c\") "
Nov 03 20:51:43.147794 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:43.147760    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53370acb-4269-4c64-947d-c8f98d5beb9c-openshift-service-ca" pod "53370acb-4269-4c64-947d-c8f98d5beb9c" (UID: "53370acb-4269-4c64-947d-c8f98d5beb9c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:51:43.149041 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:43.149016    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53370acb-4269-4c64-947d-c8f98d5beb9c-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:43.149575 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:43.149544    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53370acb-4269-4c64-947d-c8f98d5beb9c-metrics-certs" pod "53370acb-4269-4c64-947d-c8f98d5beb9c" (UID: "53370acb-4269-4c64-947d-c8f98d5beb9c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:51:43.149781 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:43.149756    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53370acb-4269-4c64-947d-c8f98d5beb9c-kube-api-access-gmdp9" pod "53370acb-4269-4c64-947d-c8f98d5beb9c" (UID: "53370acb-4269-4c64-947d-c8f98d5beb9c"). InnerVolumeSpecName "kube-api-access-gmdp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:51:43.149969 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:43.149947    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53370acb-4269-4c64-947d-c8f98d5beb9c-maas-api-tls" pod "53370acb-4269-4c64-947d-c8f98d5beb9c" (UID: "53370acb-4269-4c64-947d-c8f98d5beb9c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:51:43.215540 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:43.215497    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf" podUID="46ec9f9f-d32d-486e-8c5a-6bc074038ca7" containerName="payload-pre-processing" containerID="cri-o://7e2ffbc6ad030625ddd258a518df01a41ccb8e35ef6e787db4497b91849aa616" gracePeriod=25
Nov 03 20:51:43.276625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:43.276004    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53370acb-4269-4c64-947d-c8f98d5beb9c-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:43.276625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:43.276049    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmdp9\" (UniqueName: \"kubernetes.io/projected/53370acb-4269-4c64-947d-c8f98d5beb9c-kube-api-access-gmdp9\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:43.276625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:43.276067    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53370acb-4269-4c64-947d-c8f98d5beb9c-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:43.726680 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:43.726646    4421 generic.go:356] "Generic (PLEG): container finished" podID="53370acb-4269-4c64-947d-c8f98d5beb9c" containerID="8470cff6cb70704156ab8ad826db61c0f11f9ea439068086397c8e0ad7bd74f5" exitCode=0
Nov 03 20:51:43.727167 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:43.726697    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk"
Nov 03 20:51:43.727167 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:43.726705    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk" event={"ID":"53370acb-4269-4c64-947d-c8f98d5beb9c","Type":"ContainerDied","Data":"8470cff6cb70704156ab8ad826db61c0f11f9ea439068086397c8e0ad7bd74f5"}
Nov 03 20:51:43.727167 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:43.726764    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk" event={"ID":"53370acb-4269-4c64-947d-c8f98d5beb9c","Type":"ContainerDied","Data":"e11683f21bc125f2b19fbb7ea11dbdddcc297b25f5a8e4883045fd2a47e7420a"}
Nov 03 20:51:43.727167 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:43.726788    4421 scope.go:122] "RemoveContainer" containerID="8470cff6cb70704156ab8ad826db61c0f11f9ea439068086397c8e0ad7bd74f5"
Nov 03 20:51:43.739365 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:43.739350    4421 scope.go:122] "RemoveContainer" containerID="8470cff6cb70704156ab8ad826db61c0f11f9ea439068086397c8e0ad7bd74f5"
Nov 03 20:51:43.739689 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:51:43.739628    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8470cff6cb70704156ab8ad826db61c0f11f9ea439068086397c8e0ad7bd74f5\": container with ID starting with 8470cff6cb70704156ab8ad826db61c0f11f9ea439068086397c8e0ad7bd74f5 not found: ID does not exist" containerID="8470cff6cb70704156ab8ad826db61c0f11f9ea439068086397c8e0ad7bd74f5"
Nov 03 20:51:43.739689 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:43.739660    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8470cff6cb70704156ab8ad826db61c0f11f9ea439068086397c8e0ad7bd74f5"} err="failed to get container status \"8470cff6cb70704156ab8ad826db61c0f11f9ea439068086397c8e0ad7bd74f5\": rpc error: code = NotFound desc = could not find container \"8470cff6cb70704156ab8ad826db61c0f11f9ea439068086397c8e0ad7bd74f5\": container with ID starting with 8470cff6cb70704156ab8ad826db61c0f11f9ea439068086397c8e0ad7bd74f5 not found: ID does not exist"
Nov 03 20:51:43.753835 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:43.753806    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk"]
Nov 03 20:51:43.758272 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:43.758253    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8c6934-6756f76bf9-f5xmk"]
Nov 03 20:51:44.018765 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:44.018688    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53370acb-4269-4c64-947d-c8f98d5beb9c" path="/var/lib/kubelet/pods/53370acb-4269-4c64-947d-c8f98d5beb9c/volumes"
Nov 03 20:51:46.107097 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:46.107053    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"]
Nov 03 20:51:46.107883 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:46.107850    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km" podUID="770e86f1-0fe5-43db-b0c2-cb0c3396c8d0" containerName="istio-proxy" containerID="cri-o://92378a3429a7319100d37bdde7df2068fe09fa82705ea1b27a06a3220f4ea11b" gracePeriod=30
Nov 03 20:51:46.887078 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:46.887044    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q" podUID="4582fd72-a3a0-4d41-a3a4-29cf500022f4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:51:47.230330 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:47.230299    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv"]
Nov 03 20:51:47.231455 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:47.231431    4421 cpu_manager.go:414] "containerMap: removing container" podUID="53370acb-4269-4c64-947d-c8f98d5beb9c" containerName="maas-api"
Nov 03 20:51:47.231455 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:47.231457    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53370acb-4269-4c64-947d-c8f98d5beb9c" containerName="maas-api"
Nov 03 20:51:47.231644 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:47.231632    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="53370acb-4269-4c64-947d-c8f98d5beb9c" containerName="maas-api"
Nov 03 20:51:47.257974 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:47.257819    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv"]
Nov 03 20:51:47.257974 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:47.257941    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv"
Nov 03 20:51:47.263903 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:47.263872    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-745c52-serving-cert\""
Nov 03 20:51:47.284904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:47.284873    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8099016-08bd-44d8-9b1a-d586228375f0-metrics-certs\") pod \"maas-api-e2e-inf-a-745c52-9cc597848-p5mqv\" (UID: \"a8099016-08bd-44d8-9b1a-d586228375f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv"
Nov 03 20:51:47.285135 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:47.285115    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tn2g\" (UniqueName: \"kubernetes.io/projected/a8099016-08bd-44d8-9b1a-d586228375f0-kube-api-access-4tn2g\") pod \"maas-api-e2e-inf-a-745c52-9cc597848-p5mqv\" (UID: \"a8099016-08bd-44d8-9b1a-d586228375f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv"
Nov 03 20:51:47.285427 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:47.285403    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8099016-08bd-44d8-9b1a-d586228375f0-maas-api-tls\") pod \"maas-api-e2e-inf-a-745c52-9cc597848-p5mqv\" (UID: \"a8099016-08bd-44d8-9b1a-d586228375f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv"
Nov 03 20:51:47.285723 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:47.285451    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8099016-08bd-44d8-9b1a-d586228375f0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-745c52-9cc597848-p5mqv\" (UID: \"a8099016-08bd-44d8-9b1a-d586228375f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv"
Nov 03 20:51:47.390622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:47.390558    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8099016-08bd-44d8-9b1a-d586228375f0-metrics-certs\") pod \"maas-api-e2e-inf-a-745c52-9cc597848-p5mqv\" (UID: \"a8099016-08bd-44d8-9b1a-d586228375f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv"
Nov 03 20:51:47.390850 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:47.390641    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tn2g\" (UniqueName: \"kubernetes.io/projected/a8099016-08bd-44d8-9b1a-d586228375f0-kube-api-access-4tn2g\") pod \"maas-api-e2e-inf-a-745c52-9cc597848-p5mqv\" (UID: \"a8099016-08bd-44d8-9b1a-d586228375f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv"
Nov 03 20:51:47.390850 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:47.390840    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8099016-08bd-44d8-9b1a-d586228375f0-maas-api-tls\") pod \"maas-api-e2e-inf-a-745c52-9cc597848-p5mqv\" (UID: \"a8099016-08bd-44d8-9b1a-d586228375f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv"
Nov 03 20:51:47.391045 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:47.390890    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8099016-08bd-44d8-9b1a-d586228375f0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-745c52-9cc597848-p5mqv\" (UID: \"a8099016-08bd-44d8-9b1a-d586228375f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv"
Nov 03 20:51:47.393358 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:47.392297    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8099016-08bd-44d8-9b1a-d586228375f0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-745c52-9cc597848-p5mqv\" (UID: \"a8099016-08bd-44d8-9b1a-d586228375f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv"
Nov 03 20:51:47.397366 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:47.395808    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8099016-08bd-44d8-9b1a-d586228375f0-metrics-certs\") pod \"maas-api-e2e-inf-a-745c52-9cc597848-p5mqv\" (UID: \"a8099016-08bd-44d8-9b1a-d586228375f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv"
Nov 03 20:51:47.404470 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:47.403926    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tn2g\" (UniqueName: \"kubernetes.io/projected/a8099016-08bd-44d8-9b1a-d586228375f0-kube-api-access-4tn2g\") pod \"maas-api-e2e-inf-a-745c52-9cc597848-p5mqv\" (UID: \"a8099016-08bd-44d8-9b1a-d586228375f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv"
Nov 03 20:51:47.406846 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:47.406821    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8099016-08bd-44d8-9b1a-d586228375f0-maas-api-tls\") pod \"maas-api-e2e-inf-a-745c52-9cc597848-p5mqv\" (UID: \"a8099016-08bd-44d8-9b1a-d586228375f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv"
Nov 03 20:51:47.592380 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:47.592294    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv"
Nov 03 20:51:47.950123 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:47.942288    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv"]
Nov 03 20:51:47.950123 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:47.944357    4421 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:51:48.792956 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:48.792922    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv" event={"ID":"a8099016-08bd-44d8-9b1a-d586228375f0","Type":"ContainerStarted","Data":"6a6fafd4850bae25d390a8ae52a9a7a442182eaf57e409f65b16f92c8e79b831"}
Nov 03 20:51:48.793693 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:48.792976    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv" event={"ID":"a8099016-08bd-44d8-9b1a-d586228375f0","Type":"ContainerStarted","Data":"68f488badf2206aadfd5b1ceddf5ee74cb890cff230488a845a06e09eeab93a8"}
Nov 03 20:51:48.793693 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:48.793013    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv"
Nov 03 20:51:48.815633 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:48.815560    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv" podStartSLOduration=1.167413366 podStartE2EDuration="1.815546765s" totalImagesPullingTime="648.133399ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:48.812212946 +0000 UTC m=+3038.945341408" watchObservedRunningTime="2026-11-03 20:51:48.815546765 +0000 UTC m=+3038.948675227"
Nov 03 20:51:49.629899 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:51:49.629867    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:51:50.059922 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:50.059879    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf" podUID="46ec9f9f-d32d-486e-8c5a-6bc074038ca7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:51:51.831005 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.830977    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:51:51.834073 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.834047    4421 generic.go:356] "Generic (PLEG): container finished" podID="770e86f1-0fe5-43db-b0c2-cb0c3396c8d0" containerID="92378a3429a7319100d37bdde7df2068fe09fa82705ea1b27a06a3220f4ea11b" exitCode=0
Nov 03 20:51:51.834209 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.834091    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"
Nov 03 20:51:51.834209 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.834128    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km" event={"ID":"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0","Type":"ContainerDied","Data":"92378a3429a7319100d37bdde7df2068fe09fa82705ea1b27a06a3220f4ea11b"}
Nov 03 20:51:51.834312 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.834222    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km" event={"ID":"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0","Type":"ContainerDied","Data":"5368875989eb972238f98952d2a938aaf4aedc40eb0aac49caa5abd02266d488"}
Nov 03 20:51:51.834312 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.834263    4421 scope.go:122] "RemoveContainer" containerID="92378a3429a7319100d37bdde7df2068fe09fa82705ea1b27a06a3220f4ea11b"
Nov 03 20:51:51.845793 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.845777    4421 scope.go:122] "RemoveContainer" containerID="92378a3429a7319100d37bdde7df2068fe09fa82705ea1b27a06a3220f4ea11b"
Nov 03 20:51:51.846067 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:51:51.846048    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92378a3429a7319100d37bdde7df2068fe09fa82705ea1b27a06a3220f4ea11b\": container with ID starting with 92378a3429a7319100d37bdde7df2068fe09fa82705ea1b27a06a3220f4ea11b not found: ID does not exist" containerID="92378a3429a7319100d37bdde7df2068fe09fa82705ea1b27a06a3220f4ea11b"
Nov 03 20:51:51.846136 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.846078    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92378a3429a7319100d37bdde7df2068fe09fa82705ea1b27a06a3220f4ea11b"} err="failed to get container status \"92378a3429a7319100d37bdde7df2068fe09fa82705ea1b27a06a3220f4ea11b\": rpc error: code = NotFound desc = could not find container \"92378a3429a7319100d37bdde7df2068fe09fa82705ea1b27a06a3220f4ea11b\": container with ID starting with 92378a3429a7319100d37bdde7df2068fe09fa82705ea1b27a06a3220f4ea11b not found: ID does not exist"
Nov 03 20:51:51.876487 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.876461    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-podinfo\") pod \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") "
Nov 03 20:51:51.876698 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.876507    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-workload-socket\") pod \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") "
Nov 03 20:51:51.876698 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.876557    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-token\" (UniqueName: \"kubernetes.io/projected/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-token\") pod \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") "
Nov 03 20:51:51.876965 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.876908    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-workload-socket" pod "770e86f1-0fe5-43db-b0c2-cb0c3396c8d0" (UID: "770e86f1-0fe5-43db-b0c2-cb0c3396c8d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:51:51.877119 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.877097    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istiod-ca-cert\") pod \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") "
Nov 03 20:51:51.878466 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.878439    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istiod-ca-cert" pod "770e86f1-0fe5-43db-b0c2-cb0c3396c8d0" (UID: "770e86f1-0fe5-43db-b0c2-cb0c3396c8d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:51:51.878580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.878505    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-data\") pod \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") "
Nov 03 20:51:51.878689 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.878638    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-envoy\") pod \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") "
Nov 03 20:51:51.878791 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.878684    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-kube-api-access-nzqn9\" (UniqueName: \"kubernetes.io/projected/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-kube-api-access-nzqn9\") pod \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") "
Nov 03 20:51:51.878791 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.878728    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-credential-socket\") pod \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") "
Nov 03 20:51:51.878791 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.878786    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-workload-certs\") pod \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\" (UID: \"770e86f1-0fe5-43db-b0c2-cb0c3396c8d0\") "
Nov 03 20:51:51.880342 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.879741    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-podinfo" pod "770e86f1-0fe5-43db-b0c2-cb0c3396c8d0" (UID: "770e86f1-0fe5-43db-b0c2-cb0c3396c8d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:51:51.880342 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.879896    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:51.880342 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.879916    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:51.880342 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.879930    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:51.880342 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.880198    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-token" pod "770e86f1-0fe5-43db-b0c2-cb0c3396c8d0" (UID: "770e86f1-0fe5-43db-b0c2-cb0c3396c8d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:51:51.880342 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.880205    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-workload-certs" pod "770e86f1-0fe5-43db-b0c2-cb0c3396c8d0" (UID: "770e86f1-0fe5-43db-b0c2-cb0c3396c8d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:51:51.880342 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.880267    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-credential-socket" pod "770e86f1-0fe5-43db-b0c2-cb0c3396c8d0" (UID: "770e86f1-0fe5-43db-b0c2-cb0c3396c8d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:51:51.880342 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.880315    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-data" pod "770e86f1-0fe5-43db-b0c2-cb0c3396c8d0" (UID: "770e86f1-0fe5-43db-b0c2-cb0c3396c8d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:51:51.881796 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.881758    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-envoy" pod "770e86f1-0fe5-43db-b0c2-cb0c3396c8d0" (UID: "770e86f1-0fe5-43db-b0c2-cb0c3396c8d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:51:51.882640 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.882583    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-kube-api-access-nzqn9" pod "770e86f1-0fe5-43db-b0c2-cb0c3396c8d0" (UID: "770e86f1-0fe5-43db-b0c2-cb0c3396c8d0"). InnerVolumeSpecName "kube-api-access-nzqn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:51:51.981722 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.981691    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:51.981722 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.981718    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:51.981722 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.981727    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:51.981966 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.981737    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzqn9\" (UniqueName: \"kubernetes.io/projected/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-kube-api-access-nzqn9\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:51.981966 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.981748    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:51.981966 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:51.981756    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:51:52.155026 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:52.154946    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"]
Nov 03 20:51:52.159743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:52.159714    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8c6934-openshift-default-86d87fffbf-ms6km"]
Nov 03 20:51:52.402932 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:52.402902    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq"]
Nov 03 20:51:52.421977 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:52.421901    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm"]
Nov 03 20:51:54.026615 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:54.026572    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="770e86f1-0fe5-43db-b0c2-cb0c3396c8d0" path="/var/lib/kubelet/pods/770e86f1-0fe5-43db-b0c2-cb0c3396c8d0/volumes"
Nov 03 20:51:54.821979 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:54.821953    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv"
Nov 03 20:51:56.889448 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:56.889400    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q" podUID="4582fd72-a3a0-4d41-a3a4-29cf500022f4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:51:57.403188 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:57.403137    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq" podUID="11490efe-bbaa-4384-a3cf-bfccd7b6dca3" containerName="payload-processing" containerID="cri-o://ae90f48a96c49b8f20c462dc12ff090ea705e4350bb52c2eb877c56e35072314" gracePeriod=25
Nov 03 20:51:57.422709 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:57.422669    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm" podUID="2dec8202-6194-4d04-a73a-f9b037827f5e" containerName="payload-pre-processing" containerID="cri-o://12049298fc9471031557dc4a899d44ce496fbb2a9dc7a9ca116150156ca8a055" gracePeriod=25
Nov 03 20:51:59.240307 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:59.240272    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm" podUID="2dec8202-6194-4d04-a73a-f9b037827f5e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:51:59.240865 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:59.240322    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq" podUID="11490efe-bbaa-4384-a3cf-bfccd7b6dca3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:51:59.903947 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:59.903923    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q"
Nov 03 20:51:59.936663 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:59.936634    4421 generic.go:356] "Generic (PLEG): container finished" podID="4582fd72-a3a0-4d41-a3a4-29cf500022f4" containerID="e0172df076c1736fe2d1ec4d682fa9a9e92ed94aefea8513f9afabba321853d5" exitCode=0
Nov 03 20:51:59.936832 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:59.936673    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q"
Nov 03 20:51:59.936832 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:59.936712    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q" event={"ID":"4582fd72-a3a0-4d41-a3a4-29cf500022f4","Type":"ContainerDied","Data":"e0172df076c1736fe2d1ec4d682fa9a9e92ed94aefea8513f9afabba321853d5"}
Nov 03 20:51:59.936987 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:59.936866    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q" event={"ID":"4582fd72-a3a0-4d41-a3a4-29cf500022f4","Type":"ContainerDied","Data":"652f03a51734675cb96452e663fe2943132887bcce91372f22ab8857a3335128"}
Nov 03 20:51:59.936987 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:59.936904    4421 scope.go:122] "RemoveContainer" containerID="e0172df076c1736fe2d1ec4d682fa9a9e92ed94aefea8513f9afabba321853d5"
Nov 03 20:51:59.950862 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:59.950836    4421 scope.go:122] "RemoveContainer" containerID="e0172df076c1736fe2d1ec4d682fa9a9e92ed94aefea8513f9afabba321853d5"
Nov 03 20:51:59.951225 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:51:59.951193    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0172df076c1736fe2d1ec4d682fa9a9e92ed94aefea8513f9afabba321853d5\": container with ID starting with e0172df076c1736fe2d1ec4d682fa9a9e92ed94aefea8513f9afabba321853d5 not found: ID does not exist" containerID="e0172df076c1736fe2d1ec4d682fa9a9e92ed94aefea8513f9afabba321853d5"
Nov 03 20:51:59.951329 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:51:59.951235    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0172df076c1736fe2d1ec4d682fa9a9e92ed94aefea8513f9afabba321853d5"} err="failed to get container status \"e0172df076c1736fe2d1ec4d682fa9a9e92ed94aefea8513f9afabba321853d5\": rpc error: code = NotFound desc = could not find container \"e0172df076c1736fe2d1ec4d682fa9a9e92ed94aefea8513f9afabba321853d5\": container with ID starting with e0172df076c1736fe2d1ec4d682fa9a9e92ed94aefea8513f9afabba321853d5 not found: ID does not exist"
Nov 03 20:52:00.037165 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:00.037133    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4582fd72-a3a0-4d41-a3a4-29cf500022f4-kube-api-access-f6kwg\" (UniqueName: \"kubernetes.io/projected/4582fd72-a3a0-4d41-a3a4-29cf500022f4-kube-api-access-f6kwg\") pod \"4582fd72-a3a0-4d41-a3a4-29cf500022f4\" (UID: \"4582fd72-a3a0-4d41-a3a4-29cf500022f4\") "
Nov 03 20:52:00.037390 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:00.037241    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4582fd72-a3a0-4d41-a3a4-29cf500022f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4582fd72-a3a0-4d41-a3a4-29cf500022f4-plugins-config-volume\") pod \"4582fd72-a3a0-4d41-a3a4-29cf500022f4\" (UID: \"4582fd72-a3a0-4d41-a3a4-29cf500022f4\") "
Nov 03 20:52:00.037637 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:00.037575    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4582fd72-a3a0-4d41-a3a4-29cf500022f4-plugins-config-volume" pod "4582fd72-a3a0-4d41-a3a4-29cf500022f4" (UID: "4582fd72-a3a0-4d41-a3a4-29cf500022f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:00.038023 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:00.038002    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4582fd72-a3a0-4d41-a3a4-29cf500022f4-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:00.039590 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:00.039569    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4582fd72-a3a0-4d41-a3a4-29cf500022f4-kube-api-access-f6kwg" pod "4582fd72-a3a0-4d41-a3a4-29cf500022f4" (UID: "4582fd72-a3a0-4d41-a3a4-29cf500022f4"). InnerVolumeSpecName "kube-api-access-f6kwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:00.060461 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:00.060424    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf" podUID="46ec9f9f-d32d-486e-8c5a-6bc074038ca7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:52:00.139801 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:00.139728    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6kwg\" (UniqueName: \"kubernetes.io/projected/4582fd72-a3a0-4d41-a3a4-29cf500022f4-kube-api-access-f6kwg\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:00.264027 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:00.263995    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q"]
Nov 03 20:52:00.268576 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:00.268546    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c94fdf-65977c7758-rdb5q"]
Nov 03 20:52:02.020166 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:02.020133    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4582fd72-a3a0-4d41-a3a4-29cf500022f4" path="/var/lib/kubelet/pods/4582fd72-a3a0-4d41-a3a4-29cf500022f4/volumes"
Nov 03 20:52:03.553911 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:03.553882    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf"
Nov 03 20:52:03.703923 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:03.703890    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46ec9f9f-d32d-486e-8c5a-6bc074038ca7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46ec9f9f-d32d-486e-8c5a-6bc074038ca7-plugins-config-volume\") pod \"46ec9f9f-d32d-486e-8c5a-6bc074038ca7\" (UID: \"46ec9f9f-d32d-486e-8c5a-6bc074038ca7\") "
Nov 03 20:52:03.704121 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:03.703962    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ec9f9f-d32d-486e-8c5a-6bc074038ca7-kube-api-access-7w42f\" (UniqueName: \"kubernetes.io/projected/46ec9f9f-d32d-486e-8c5a-6bc074038ca7-kube-api-access-7w42f\") pod \"46ec9f9f-d32d-486e-8c5a-6bc074038ca7\" (UID: \"46ec9f9f-d32d-486e-8c5a-6bc074038ca7\") "
Nov 03 20:52:03.711870 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:03.711833    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ec9f9f-d32d-486e-8c5a-6bc074038ca7-plugins-config-volume" pod "46ec9f9f-d32d-486e-8c5a-6bc074038ca7" (UID: "46ec9f9f-d32d-486e-8c5a-6bc074038ca7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:03.717758 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:03.717711    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ec9f9f-d32d-486e-8c5a-6bc074038ca7-kube-api-access-7w42f" pod "46ec9f9f-d32d-486e-8c5a-6bc074038ca7" (UID: "46ec9f9f-d32d-486e-8c5a-6bc074038ca7"). InnerVolumeSpecName "kube-api-access-7w42f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:03.806754 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:03.806672    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46ec9f9f-d32d-486e-8c5a-6bc074038ca7-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:03.806754 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:03.806702    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w42f\" (UniqueName: \"kubernetes.io/projected/46ec9f9f-d32d-486e-8c5a-6bc074038ca7-kube-api-access-7w42f\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:03.992180 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:03.992139    4421 generic.go:356] "Generic (PLEG): container finished" podID="46ec9f9f-d32d-486e-8c5a-6bc074038ca7" containerID="7e2ffbc6ad030625ddd258a518df01a41ccb8e35ef6e787db4497b91849aa616" exitCode=0
Nov 03 20:52:03.992361 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:03.992192    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf" event={"ID":"46ec9f9f-d32d-486e-8c5a-6bc074038ca7","Type":"ContainerDied","Data":"7e2ffbc6ad030625ddd258a518df01a41ccb8e35ef6e787db4497b91849aa616"}
Nov 03 20:52:03.992361 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:03.992213    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf"
Nov 03 20:52:03.992361 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:03.992261    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf" event={"ID":"46ec9f9f-d32d-486e-8c5a-6bc074038ca7","Type":"ContainerDied","Data":"8c8a46230ec9d6fabd837f3e3a83ca9cf9231617b10c4046b25f195bfcb67fb1"}
Nov 03 20:52:03.992361 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:03.992286    4421 scope.go:122] "RemoveContainer" containerID="7e2ffbc6ad030625ddd258a518df01a41ccb8e35ef6e787db4497b91849aa616"
Nov 03 20:52:04.004273 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:04.004250    4421 scope.go:122] "RemoveContainer" containerID="7e2ffbc6ad030625ddd258a518df01a41ccb8e35ef6e787db4497b91849aa616"
Nov 03 20:52:04.004536 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:52:04.004518    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e2ffbc6ad030625ddd258a518df01a41ccb8e35ef6e787db4497b91849aa616\": container with ID starting with 7e2ffbc6ad030625ddd258a518df01a41ccb8e35ef6e787db4497b91849aa616 not found: ID does not exist" containerID="7e2ffbc6ad030625ddd258a518df01a41ccb8e35ef6e787db4497b91849aa616"
Nov 03 20:52:04.004593 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:04.004544    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e2ffbc6ad030625ddd258a518df01a41ccb8e35ef6e787db4497b91849aa616"} err="failed to get container status \"7e2ffbc6ad030625ddd258a518df01a41ccb8e35ef6e787db4497b91849aa616\": rpc error: code = NotFound desc = could not find container \"7e2ffbc6ad030625ddd258a518df01a41ccb8e35ef6e787db4497b91849aa616\": container with ID starting with 7e2ffbc6ad030625ddd258a518df01a41ccb8e35ef6e787db4497b91849aa616 not found: ID does not exist"
Nov 03 20:52:04.020568 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:04.020538    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf"]
Nov 03 20:52:04.024642 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:04.024618    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8c6934-587f859b95-ggbjf"]
Nov 03 20:52:06.017904 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:06.017870    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46ec9f9f-d32d-486e-8c5a-6bc074038ca7" path="/var/lib/kubelet/pods/46ec9f9f-d32d-486e-8c5a-6bc074038ca7/volumes"
Nov 03 20:52:09.240624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:09.240031    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq" podUID="11490efe-bbaa-4384-a3cf-bfccd7b6dca3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:52:09.240624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:09.240503    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm" podUID="2dec8202-6194-4d04-a73a-f9b037827f5e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:52:16.434685 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:16.434648    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864"]
Nov 03 20:52:16.435081 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:16.434915    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864" podUID="0cf2df26-0976-4da0-b3fa-2b036625fc6e" containerName="maas-api" containerID="cri-o://bb5d5d9e2a6417eae37d80a4504f5ee5826b5edcc25db5fb7f6c211fdd05ade9" gracePeriod=30
Nov 03 20:52:17.030527 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.030500    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864"
Nov 03 20:52:17.153879 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.153846    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cf2df26-0976-4da0-b3fa-2b036625fc6e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf2df26-0976-4da0-b3fa-2b036625fc6e-openshift-service-ca\") pod \"0cf2df26-0976-4da0-b3fa-2b036625fc6e\" (UID: \"0cf2df26-0976-4da0-b3fa-2b036625fc6e\") "
Nov 03 20:52:17.154051 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.153884    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cf2df26-0976-4da0-b3fa-2b036625fc6e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cf2df26-0976-4da0-b3fa-2b036625fc6e-metrics-certs\") pod \"0cf2df26-0976-4da0-b3fa-2b036625fc6e\" (UID: \"0cf2df26-0976-4da0-b3fa-2b036625fc6e\") "
Nov 03 20:52:17.154151 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.154129    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cf2df26-0976-4da0-b3fa-2b036625fc6e-kube-api-access-nwmzg\" (UniqueName: \"kubernetes.io/projected/0cf2df26-0976-4da0-b3fa-2b036625fc6e-kube-api-access-nwmzg\") pod \"0cf2df26-0976-4da0-b3fa-2b036625fc6e\" (UID: \"0cf2df26-0976-4da0-b3fa-2b036625fc6e\") "
Nov 03 20:52:17.154256 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.154232    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cf2df26-0976-4da0-b3fa-2b036625fc6e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cf2df26-0976-4da0-b3fa-2b036625fc6e-maas-api-tls\") pod \"0cf2df26-0976-4da0-b3fa-2b036625fc6e\" (UID: \"0cf2df26-0976-4da0-b3fa-2b036625fc6e\") "
Nov 03 20:52:17.154256 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.154244    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf2df26-0976-4da0-b3fa-2b036625fc6e-openshift-service-ca" pod "0cf2df26-0976-4da0-b3fa-2b036625fc6e" (UID: "0cf2df26-0976-4da0-b3fa-2b036625fc6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:17.155115 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.155090    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf2df26-0976-4da0-b3fa-2b036625fc6e-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:17.156237 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.156204    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cf2df26-0976-4da0-b3fa-2b036625fc6e-metrics-certs" pod "0cf2df26-0976-4da0-b3fa-2b036625fc6e" (UID: "0cf2df26-0976-4da0-b3fa-2b036625fc6e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:52:17.156537 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.156507    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cf2df26-0976-4da0-b3fa-2b036625fc6e-maas-api-tls" pod "0cf2df26-0976-4da0-b3fa-2b036625fc6e" (UID: "0cf2df26-0976-4da0-b3fa-2b036625fc6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:52:17.156671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.156643    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf2df26-0976-4da0-b3fa-2b036625fc6e-kube-api-access-nwmzg" pod "0cf2df26-0976-4da0-b3fa-2b036625fc6e" (UID: "0cf2df26-0976-4da0-b3fa-2b036625fc6e"). InnerVolumeSpecName "kube-api-access-nwmzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:17.257351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.257143    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwmzg\" (UniqueName: \"kubernetes.io/projected/0cf2df26-0976-4da0-b3fa-2b036625fc6e-kube-api-access-nwmzg\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:17.257351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.257180    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cf2df26-0976-4da0-b3fa-2b036625fc6e-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:17.257351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.257194    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cf2df26-0976-4da0-b3fa-2b036625fc6e-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:17.284019 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.283973    4421 generic.go:356] "Generic (PLEG): container finished" podID="0cf2df26-0976-4da0-b3fa-2b036625fc6e" containerID="bb5d5d9e2a6417eae37d80a4504f5ee5826b5edcc25db5fb7f6c211fdd05ade9" exitCode=0
Nov 03 20:52:17.284184 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.284049    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864"
Nov 03 20:52:17.284184 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.284045    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864" event={"ID":"0cf2df26-0976-4da0-b3fa-2b036625fc6e","Type":"ContainerDied","Data":"bb5d5d9e2a6417eae37d80a4504f5ee5826b5edcc25db5fb7f6c211fdd05ade9"}
Nov 03 20:52:17.284184 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.284143    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864" event={"ID":"0cf2df26-0976-4da0-b3fa-2b036625fc6e","Type":"ContainerDied","Data":"809138cd119c62a56cd9f65e52edf48b5796949a4d5c555bd6f441286926abd0"}
Nov 03 20:52:17.284184 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.284171    4421 scope.go:122] "RemoveContainer" containerID="bb5d5d9e2a6417eae37d80a4504f5ee5826b5edcc25db5fb7f6c211fdd05ade9"
Nov 03 20:52:17.300005 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.299984    4421 scope.go:122] "RemoveContainer" containerID="bb5d5d9e2a6417eae37d80a4504f5ee5826b5edcc25db5fb7f6c211fdd05ade9"
Nov 03 20:52:17.300306 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:52:17.300288    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb5d5d9e2a6417eae37d80a4504f5ee5826b5edcc25db5fb7f6c211fdd05ade9\": container with ID starting with bb5d5d9e2a6417eae37d80a4504f5ee5826b5edcc25db5fb7f6c211fdd05ade9 not found: ID does not exist" containerID="bb5d5d9e2a6417eae37d80a4504f5ee5826b5edcc25db5fb7f6c211fdd05ade9"
Nov 03 20:52:17.300353 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.300317    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb5d5d9e2a6417eae37d80a4504f5ee5826b5edcc25db5fb7f6c211fdd05ade9"} err="failed to get container status \"bb5d5d9e2a6417eae37d80a4504f5ee5826b5edcc25db5fb7f6c211fdd05ade9\": rpc error: code = NotFound desc = could not find container \"bb5d5d9e2a6417eae37d80a4504f5ee5826b5edcc25db5fb7f6c211fdd05ade9\": container with ID starting with bb5d5d9e2a6417eae37d80a4504f5ee5826b5edcc25db5fb7f6c211fdd05ade9 not found: ID does not exist"
Nov 03 20:52:17.313889 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.313858    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864"]
Nov 03 20:52:17.318457 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.318436    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f5afd7-7d5b6fb664-l4864"]
Nov 03 20:52:17.743655 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.743627    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq"
Nov 03 20:52:17.842245 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.837763    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm"
Nov 03 20:52:17.925947 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.925915    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11490efe-bbaa-4384-a3cf-bfccd7b6dca3-kube-api-access-bmmjh\" (UniqueName: \"kubernetes.io/projected/11490efe-bbaa-4384-a3cf-bfccd7b6dca3-kube-api-access-bmmjh\") pod \"11490efe-bbaa-4384-a3cf-bfccd7b6dca3\" (UID: \"11490efe-bbaa-4384-a3cf-bfccd7b6dca3\") "
Nov 03 20:52:17.926128 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.926027    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11490efe-bbaa-4384-a3cf-bfccd7b6dca3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11490efe-bbaa-4384-a3cf-bfccd7b6dca3-plugins-config-volume\") pod \"11490efe-bbaa-4384-a3cf-bfccd7b6dca3\" (UID: \"11490efe-bbaa-4384-a3cf-bfccd7b6dca3\") "
Nov 03 20:52:17.926409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.926390    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11490efe-bbaa-4384-a3cf-bfccd7b6dca3-plugins-config-volume" pod "11490efe-bbaa-4384-a3cf-bfccd7b6dca3" (UID: "11490efe-bbaa-4384-a3cf-bfccd7b6dca3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:17.926767 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.926749    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11490efe-bbaa-4384-a3cf-bfccd7b6dca3-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:17.928260 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:17.928242    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11490efe-bbaa-4384-a3cf-bfccd7b6dca3-kube-api-access-bmmjh" pod "11490efe-bbaa-4384-a3cf-bfccd7b6dca3" (UID: "11490efe-bbaa-4384-a3cf-bfccd7b6dca3"). InnerVolumeSpecName "kube-api-access-bmmjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:18.018927 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:18.018895    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cf2df26-0976-4da0-b3fa-2b036625fc6e" path="/var/lib/kubelet/pods/0cf2df26-0976-4da0-b3fa-2b036625fc6e/volumes"
Nov 03 20:52:18.027718 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:18.027692    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dec8202-6194-4d04-a73a-f9b037827f5e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dec8202-6194-4d04-a73a-f9b037827f5e-plugins-config-volume\") pod \"2dec8202-6194-4d04-a73a-f9b037827f5e\" (UID: \"2dec8202-6194-4d04-a73a-f9b037827f5e\") "
Nov 03 20:52:18.027892 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:18.027757    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dec8202-6194-4d04-a73a-f9b037827f5e-kube-api-access-g56ld\" (UniqueName: \"kubernetes.io/projected/2dec8202-6194-4d04-a73a-f9b037827f5e-kube-api-access-g56ld\") pod \"2dec8202-6194-4d04-a73a-f9b037827f5e\" (UID: \"2dec8202-6194-4d04-a73a-f9b037827f5e\") "
Nov 03 20:52:18.028108 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:18.028083    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dec8202-6194-4d04-a73a-f9b037827f5e-plugins-config-volume" pod "2dec8202-6194-4d04-a73a-f9b037827f5e" (UID: "2dec8202-6194-4d04-a73a-f9b037827f5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:18.028479 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:18.028448    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dec8202-6194-4d04-a73a-f9b037827f5e-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:18.028562 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:18.028482    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmmjh\" (UniqueName: \"kubernetes.io/projected/11490efe-bbaa-4384-a3cf-bfccd7b6dca3-kube-api-access-bmmjh\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:18.030692 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:18.030663    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dec8202-6194-4d04-a73a-f9b037827f5e-kube-api-access-g56ld" pod "2dec8202-6194-4d04-a73a-f9b037827f5e" (UID: "2dec8202-6194-4d04-a73a-f9b037827f5e"). InnerVolumeSpecName "kube-api-access-g56ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:18.131042 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:18.130969    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g56ld\" (UniqueName: \"kubernetes.io/projected/2dec8202-6194-4d04-a73a-f9b037827f5e-kube-api-access-g56ld\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:18.296089 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:18.296058    4421 generic.go:356] "Generic (PLEG): container finished" podID="11490efe-bbaa-4384-a3cf-bfccd7b6dca3" containerID="ae90f48a96c49b8f20c462dc12ff090ea705e4350bb52c2eb877c56e35072314" exitCode=0
Nov 03 20:52:18.296290 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:18.296106    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq"
Nov 03 20:52:18.296290 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:18.296142    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq" event={"ID":"11490efe-bbaa-4384-a3cf-bfccd7b6dca3","Type":"ContainerDied","Data":"ae90f48a96c49b8f20c462dc12ff090ea705e4350bb52c2eb877c56e35072314"}
Nov 03 20:52:18.296290 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:18.296194    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq" event={"ID":"11490efe-bbaa-4384-a3cf-bfccd7b6dca3","Type":"ContainerDied","Data":"3bb0c988504d574bd1a783ca2f97aaa372a4c8a6cd0beef67ee99318cff8a4e8"}
Nov 03 20:52:18.296290 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:18.296226    4421 scope.go:122] "RemoveContainer" containerID="ae90f48a96c49b8f20c462dc12ff090ea705e4350bb52c2eb877c56e35072314"
Nov 03 20:52:18.297644 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:18.297587    4421 generic.go:356] "Generic (PLEG): container finished" podID="2dec8202-6194-4d04-a73a-f9b037827f5e" containerID="12049298fc9471031557dc4a899d44ce496fbb2a9dc7a9ca116150156ca8a055" exitCode=0
Nov 03 20:52:18.297770 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:18.297662    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm"
Nov 03 20:52:18.297770 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:18.297694    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm" event={"ID":"2dec8202-6194-4d04-a73a-f9b037827f5e","Type":"ContainerDied","Data":"12049298fc9471031557dc4a899d44ce496fbb2a9dc7a9ca116150156ca8a055"}
Nov 03 20:52:18.297770 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:18.297733    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm" event={"ID":"2dec8202-6194-4d04-a73a-f9b037827f5e","Type":"ContainerDied","Data":"531e32ea4d45b38d537182a409ac1c75467d46c330194b6ee7c0b22945347ac3"}
Nov 03 20:52:18.309125 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:18.309107    4421 scope.go:122] "RemoveContainer" containerID="ae90f48a96c49b8f20c462dc12ff090ea705e4350bb52c2eb877c56e35072314"
Nov 03 20:52:18.309386 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:52:18.309364    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae90f48a96c49b8f20c462dc12ff090ea705e4350bb52c2eb877c56e35072314\": container with ID starting with ae90f48a96c49b8f20c462dc12ff090ea705e4350bb52c2eb877c56e35072314 not found: ID does not exist" containerID="ae90f48a96c49b8f20c462dc12ff090ea705e4350bb52c2eb877c56e35072314"
Nov 03 20:52:18.309458 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:18.309394    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae90f48a96c49b8f20c462dc12ff090ea705e4350bb52c2eb877c56e35072314"} err="failed to get container status \"ae90f48a96c49b8f20c462dc12ff090ea705e4350bb52c2eb877c56e35072314\": rpc error: code = NotFound desc = could not find container \"ae90f48a96c49b8f20c462dc12ff090ea705e4350bb52c2eb877c56e35072314\": container with ID starting with ae90f48a96c49b8f20c462dc12ff090ea705e4350bb52c2eb877c56e35072314 not found: ID does not exist"
Nov 03 20:52:18.309458 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:18.309411    4421 scope.go:122] "RemoveContainer" containerID="12049298fc9471031557dc4a899d44ce496fbb2a9dc7a9ca116150156ca8a055"
Nov 03 20:52:18.320475 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:18.320443    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq"]
Nov 03 20:52:18.322501 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:18.322480    4421 scope.go:122] "RemoveContainer" containerID="12049298fc9471031557dc4a899d44ce496fbb2a9dc7a9ca116150156ca8a055"
Nov 03 20:52:18.323483 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:52:18.323449    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12049298fc9471031557dc4a899d44ce496fbb2a9dc7a9ca116150156ca8a055\": container with ID starting with 12049298fc9471031557dc4a899d44ce496fbb2a9dc7a9ca116150156ca8a055 not found: ID does not exist" containerID="12049298fc9471031557dc4a899d44ce496fbb2a9dc7a9ca116150156ca8a055"
Nov 03 20:52:18.323579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:18.323482    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12049298fc9471031557dc4a899d44ce496fbb2a9dc7a9ca116150156ca8a055"} err="failed to get container status \"12049298fc9471031557dc4a899d44ce496fbb2a9dc7a9ca116150156ca8a055\": rpc error: code = NotFound desc = could not find container \"12049298fc9471031557dc4a899d44ce496fbb2a9dc7a9ca116150156ca8a055\": container with ID starting with 12049298fc9471031557dc4a899d44ce496fbb2a9dc7a9ca116150156ca8a055 not found: ID does not exist"
Nov 03 20:52:18.327149 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:18.327130    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f5afd7-5b66cf75dd-bfpsq"]
Nov 03 20:52:18.341675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:18.341654    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm"]
Nov 03 20:52:18.345713 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:18.345689    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f5afd7-f649d9cb7-9xtxm"]
Nov 03 20:52:19.630251 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:52:19.630215    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:52:19.828915 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.828881    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k"]
Nov 03 20:52:19.829857 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.829836    4421 cpu_manager.go:414] "containerMap: removing container" podUID="11490efe-bbaa-4384-a3cf-bfccd7b6dca3" containerName="payload-processing"
Nov 03 20:52:19.829857 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.829854    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11490efe-bbaa-4384-a3cf-bfccd7b6dca3" containerName="payload-processing"
Nov 03 20:52:19.830023 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.829879    4421 cpu_manager.go:414] "containerMap: removing container" podUID="46ec9f9f-d32d-486e-8c5a-6bc074038ca7" containerName="payload-pre-processing"
Nov 03 20:52:19.830023 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.829884    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ec9f9f-d32d-486e-8c5a-6bc074038ca7" containerName="payload-pre-processing"
Nov 03 20:52:19.830023 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.829897    4421 cpu_manager.go:414] "containerMap: removing container" podUID="4582fd72-a3a0-4d41-a3a4-29cf500022f4" containerName="payload-processing"
Nov 03 20:52:19.830023 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.829902    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4582fd72-a3a0-4d41-a3a4-29cf500022f4" containerName="payload-processing"
Nov 03 20:52:19.830023 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.829914    4421 cpu_manager.go:414] "containerMap: removing container" podUID="2dec8202-6194-4d04-a73a-f9b037827f5e" containerName="payload-pre-processing"
Nov 03 20:52:19.830023 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.829919    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dec8202-6194-4d04-a73a-f9b037827f5e" containerName="payload-pre-processing"
Nov 03 20:52:19.830023 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.829937    4421 cpu_manager.go:414] "containerMap: removing container" podUID="0cf2df26-0976-4da0-b3fa-2b036625fc6e" containerName="maas-api"
Nov 03 20:52:19.830023 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.829942    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cf2df26-0976-4da0-b3fa-2b036625fc6e" containerName="maas-api"
Nov 03 20:52:19.830023 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.829953    4421 cpu_manager.go:414] "containerMap: removing container" podUID="770e86f1-0fe5-43db-b0c2-cb0c3396c8d0" containerName="istio-proxy"
Nov 03 20:52:19.830023 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.829957    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="770e86f1-0fe5-43db-b0c2-cb0c3396c8d0" containerName="istio-proxy"
Nov 03 20:52:19.830351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.830081    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="11490efe-bbaa-4384-a3cf-bfccd7b6dca3" containerName="payload-processing"
Nov 03 20:52:19.830351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.830095    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ec9f9f-d32d-486e-8c5a-6bc074038ca7" containerName="payload-pre-processing"
Nov 03 20:52:19.830351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.830103    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="4582fd72-a3a0-4d41-a3a4-29cf500022f4" containerName="payload-processing"
Nov 03 20:52:19.830351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.830110    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dec8202-6194-4d04-a73a-f9b037827f5e" containerName="payload-pre-processing"
Nov 03 20:52:19.830351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.830120    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cf2df26-0976-4da0-b3fa-2b036625fc6e" containerName="maas-api"
Nov 03 20:52:19.830351 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.830126    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="770e86f1-0fe5-43db-b0c2-cb0c3396c8d0" containerName="istio-proxy"
Nov 03 20:52:19.835569 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.835551    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k"
Nov 03 20:52:19.840686 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.840659    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5ecee8\""
Nov 03 20:52:19.840952 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.840928    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5ecee8-dockercfg-hmx2v\""
Nov 03 20:52:19.845415 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.845393    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k"]
Nov 03 20:52:19.862310 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.862270    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78"]
Nov 03 20:52:19.868589 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.868566    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78"
Nov 03 20:52:19.877662 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.877584    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78"]
Nov 03 20:52:19.878241 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.877783    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x87ng\" (UniqueName: \"kubernetes.io/projected/53001e2c-5c08-4ad6-af62-053a80639a3f-kube-api-access-x87ng\") pod \"payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78\" (UID: \"53001e2c-5c08-4ad6-af62-053a80639a3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78"
Nov 03 20:52:19.878241 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.877909    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv9mw\" (UniqueName: \"kubernetes.io/projected/a12a350f-f3a6-4f34-b594-a29f2da298f4-kube-api-access-nv9mw\") pod \"payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k\" (UID: \"a12a350f-f3a6-4f34-b594-a29f2da298f4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k"
Nov 03 20:52:19.878241 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.877948    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a12a350f-f3a6-4f34-b594-a29f2da298f4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k\" (UID: \"a12a350f-f3a6-4f34-b594-a29f2da298f4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k"
Nov 03 20:52:19.878241 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.878003    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53001e2c-5c08-4ad6-af62-053a80639a3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78\" (UID: \"53001e2c-5c08-4ad6-af62-053a80639a3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78"
Nov 03 20:52:19.979564 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.979533    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x87ng\" (UniqueName: \"kubernetes.io/projected/53001e2c-5c08-4ad6-af62-053a80639a3f-kube-api-access-x87ng\") pod \"payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78\" (UID: \"53001e2c-5c08-4ad6-af62-053a80639a3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78"
Nov 03 20:52:19.979762 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.979638    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv9mw\" (UniqueName: \"kubernetes.io/projected/a12a350f-f3a6-4f34-b594-a29f2da298f4-kube-api-access-nv9mw\") pod \"payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k\" (UID: \"a12a350f-f3a6-4f34-b594-a29f2da298f4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k"
Nov 03 20:52:19.979762 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.979671    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a12a350f-f3a6-4f34-b594-a29f2da298f4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k\" (UID: \"a12a350f-f3a6-4f34-b594-a29f2da298f4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k"
Nov 03 20:52:19.979762 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.979716    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53001e2c-5c08-4ad6-af62-053a80639a3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78\" (UID: \"53001e2c-5c08-4ad6-af62-053a80639a3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78"
Nov 03 20:52:19.980446 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.980419    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a12a350f-f3a6-4f34-b594-a29f2da298f4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k\" (UID: \"a12a350f-f3a6-4f34-b594-a29f2da298f4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k"
Nov 03 20:52:19.980446 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.980445    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53001e2c-5c08-4ad6-af62-053a80639a3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78\" (UID: \"53001e2c-5c08-4ad6-af62-053a80639a3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78"
Nov 03 20:52:19.991866 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.991839    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x87ng\" (UniqueName: \"kubernetes.io/projected/53001e2c-5c08-4ad6-af62-053a80639a3f-kube-api-access-x87ng\") pod \"payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78\" (UID: \"53001e2c-5c08-4ad6-af62-053a80639a3f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78"
Nov 03 20:52:19.991990 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:19.991891    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv9mw\" (UniqueName: \"kubernetes.io/projected/a12a350f-f3a6-4f34-b594-a29f2da298f4-kube-api-access-nv9mw\") pod \"payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k\" (UID: \"a12a350f-f3a6-4f34-b594-a29f2da298f4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k"
Nov 03 20:52:20.021664 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:20.021631    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11490efe-bbaa-4384-a3cf-bfccd7b6dca3" path="/var/lib/kubelet/pods/11490efe-bbaa-4384-a3cf-bfccd7b6dca3/volumes"
Nov 03 20:52:20.022163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:20.022141    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dec8202-6194-4d04-a73a-f9b037827f5e" path="/var/lib/kubelet/pods/2dec8202-6194-4d04-a73a-f9b037827f5e/volumes"
Nov 03 20:52:20.159830 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:20.159745    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k"
Nov 03 20:52:20.228010 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:20.227977    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78"
Nov 03 20:52:20.589775 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:20.589745    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k"]
Nov 03 20:52:20.590831 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:52:20.590796    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda12a350f_f3a6_4f34_b594_a29f2da298f4.slice/crio-07ad98b9d744246147194a28790fbd1dfe7a572631f4756bdce83a0ff9fa6c52 WatchSource:0}: Error finding container 07ad98b9d744246147194a28790fbd1dfe7a572631f4756bdce83a0ff9fa6c52: Status 404 returned error can't find the container with id 07ad98b9d744246147194a28790fbd1dfe7a572631f4756bdce83a0ff9fa6c52
Nov 03 20:52:20.604926 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:20.604902    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78"]
Nov 03 20:52:20.606766 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:52:20.606740    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53001e2c_5c08_4ad6_af62_053a80639a3f.slice/crio-93cd764c50a10ac5176ffe6f3e9fa7b20ba9668cded20e2d7e69f46c497b04b2 WatchSource:0}: Error finding container 93cd764c50a10ac5176ffe6f3e9fa7b20ba9668cded20e2d7e69f46c497b04b2: Status 404 returned error can't find the container with id 93cd764c50a10ac5176ffe6f3e9fa7b20ba9668cded20e2d7e69f46c497b04b2
Nov 03 20:52:21.337889 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:21.337850    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k" event={"ID":"a12a350f-f3a6-4f34-b594-a29f2da298f4","Type":"ContainerStarted","Data":"998ba0fcbc7201b02e43ca512c202e2fbf191a93d6e9f2d151bf6a8db8466ba8"}
Nov 03 20:52:21.338386 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:21.337918    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k"
Nov 03 20:52:21.338386 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:21.337945    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k" event={"ID":"a12a350f-f3a6-4f34-b594-a29f2da298f4","Type":"ContainerStarted","Data":"07ad98b9d744246147194a28790fbd1dfe7a572631f4756bdce83a0ff9fa6c52"}
Nov 03 20:52:21.339886 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:21.339856    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78" event={"ID":"53001e2c-5c08-4ad6-af62-053a80639a3f","Type":"ContainerStarted","Data":"b70137e914fc9269b5969e54b2dd65de1bd2316563fe5c1a22d775e8787a8861"}
Nov 03 20:52:21.340024 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:21.339912    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78" event={"ID":"53001e2c-5c08-4ad6-af62-053a80639a3f","Type":"ContainerStarted","Data":"93cd764c50a10ac5176ffe6f3e9fa7b20ba9668cded20e2d7e69f46c497b04b2"}
Nov 03 20:52:21.340024 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:21.339951    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78"
Nov 03 20:52:21.358142 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:21.358096    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k" podStartSLOduration=2.358083642 podStartE2EDuration="2.358083642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:21.355807737 +0000 UTC m=+3071.488936200" watchObservedRunningTime="2026-11-03 20:52:21.358083642 +0000 UTC m=+3071.491212104"
Nov 03 20:52:21.373582 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:21.373513    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78" podStartSLOduration=2.373502498 podStartE2EDuration="2.373502498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:21.373447295 +0000 UTC m=+3071.506575757" watchObservedRunningTime="2026-11-03 20:52:21.373502498 +0000 UTC m=+3071.506630959"
Nov 03 20:52:32.389457 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:32.389429    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78"
Nov 03 20:52:32.390032 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:52:32.389502    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k"
Nov 03 20:52:49.630037 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:52:49.629951    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:53:19.633382 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:53:19.633338    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:53:20.455987 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:20.455958    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"]
Nov 03 20:53:20.456193 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:20.456174    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4" podUID="29af63ab-34a3-4ee3-9a74-b8108503642e" containerName="istio-proxy" containerID="cri-o://299b38251681a7014b217ca8e00e3aac38bf96e66492f84cd2c6a9ad89ac04ab" gracePeriod=30
Nov 03 20:53:25.772035 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.772006    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:53:25.799515 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.799483    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-workload-certs\") pod \"29af63ab-34a3-4ee3-9a74-b8108503642e\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") "
Nov 03 20:53:25.799786 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.799547    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29af63ab-34a3-4ee3-9a74-b8108503642e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29af63ab-34a3-4ee3-9a74-b8108503642e-istiod-ca-cert\") pod \"29af63ab-34a3-4ee3-9a74-b8108503642e\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") "
Nov 03 20:53:25.799786 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.799745    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-workload-socket\") pod \"29af63ab-34a3-4ee3-9a74-b8108503642e\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") "
Nov 03 20:53:25.799980 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.799959    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-credential-socket\") pod \"29af63ab-34a3-4ee3-9a74-b8108503642e\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") "
Nov 03 20:53:25.800060 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.800013    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-podinfo\") pod \"29af63ab-34a3-4ee3-9a74-b8108503642e\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") "
Nov 03 20:53:25.800060 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.800011    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-workload-socket" pod "29af63ab-34a3-4ee3-9a74-b8108503642e" (UID: "29af63ab-34a3-4ee3-9a74-b8108503642e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:53:25.800758 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.800253    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-envoy\") pod \"29af63ab-34a3-4ee3-9a74-b8108503642e\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") "
Nov 03 20:53:25.800758 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.800274    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-credential-socket" pod "29af63ab-34a3-4ee3-9a74-b8108503642e" (UID: "29af63ab-34a3-4ee3-9a74-b8108503642e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:53:25.800758 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.800290    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-token\" (UniqueName: \"kubernetes.io/projected/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-token\") pod \"29af63ab-34a3-4ee3-9a74-b8108503642e\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") "
Nov 03 20:53:25.800758 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.800332    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-data\") pod \"29af63ab-34a3-4ee3-9a74-b8108503642e\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") "
Nov 03 20:53:25.800758 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.800392    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29af63ab-34a3-4ee3-9a74-b8108503642e-kube-api-access-n4mz9\" (UniqueName: \"kubernetes.io/projected/29af63ab-34a3-4ee3-9a74-b8108503642e-kube-api-access-n4mz9\") pod \"29af63ab-34a3-4ee3-9a74-b8108503642e\" (UID: \"29af63ab-34a3-4ee3-9a74-b8108503642e\") "
Nov 03 20:53:25.800758 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.800674    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29af63ab-34a3-4ee3-9a74-b8108503642e-istiod-ca-cert" pod "29af63ab-34a3-4ee3-9a74-b8108503642e" (UID: "29af63ab-34a3-4ee3-9a74-b8108503642e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:53:25.802238 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.801880    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-data" pod "29af63ab-34a3-4ee3-9a74-b8108503642e" (UID: "29af63ab-34a3-4ee3-9a74-b8108503642e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:53:25.802238 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.801978    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29af63ab-34a3-4ee3-9a74-b8108503642e-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:25.802238 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.802034    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:25.802238 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.802054    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:25.802238 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.802207    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-workload-certs" pod "29af63ab-34a3-4ee3-9a74-b8108503642e" (UID: "29af63ab-34a3-4ee3-9a74-b8108503642e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:53:25.804088 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.804049    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-token" pod "29af63ab-34a3-4ee3-9a74-b8108503642e" (UID: "29af63ab-34a3-4ee3-9a74-b8108503642e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:53:25.804913 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.804886    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-envoy" pod "29af63ab-34a3-4ee3-9a74-b8108503642e" (UID: "29af63ab-34a3-4ee3-9a74-b8108503642e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:53:25.805030 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.804915    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-podinfo" pod "29af63ab-34a3-4ee3-9a74-b8108503642e" (UID: "29af63ab-34a3-4ee3-9a74-b8108503642e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:53:25.805515 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.805486    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29af63ab-34a3-4ee3-9a74-b8108503642e-kube-api-access-n4mz9" pod "29af63ab-34a3-4ee3-9a74-b8108503642e" (UID: "29af63ab-34a3-4ee3-9a74-b8108503642e"). InnerVolumeSpecName "kube-api-access-n4mz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:53:25.908873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.908838    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4mz9\" (UniqueName: \"kubernetes.io/projected/29af63ab-34a3-4ee3-9a74-b8108503642e-kube-api-access-n4mz9\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:25.908873 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.908874    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:25.909173 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.908887    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:25.909173 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.908900    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:25.909173 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.908912    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:25.909173 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:25.908923    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29af63ab-34a3-4ee3-9a74-b8108503642e-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:26.263320 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:26.261516    4421 generic.go:356] "Generic (PLEG): container finished" podID="29af63ab-34a3-4ee3-9a74-b8108503642e" containerID="299b38251681a7014b217ca8e00e3aac38bf96e66492f84cd2c6a9ad89ac04ab" exitCode=0
Nov 03 20:53:26.263320 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:26.261682    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4" event={"ID":"29af63ab-34a3-4ee3-9a74-b8108503642e","Type":"ContainerDied","Data":"299b38251681a7014b217ca8e00e3aac38bf96e66492f84cd2c6a9ad89ac04ab"}
Nov 03 20:53:26.263320 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:26.261731    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4" event={"ID":"29af63ab-34a3-4ee3-9a74-b8108503642e","Type":"ContainerDied","Data":"33c6d14bfe9fb032bcb33cb52ddbd86a80335815835f51ab33e0f63e1ca7785c"}
Nov 03 20:53:26.263320 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:26.261759    4421 scope.go:122] "RemoveContainer" containerID="299b38251681a7014b217ca8e00e3aac38bf96e66492f84cd2c6a9ad89ac04ab"
Nov 03 20:53:26.264671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:26.264640    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"
Nov 03 20:53:26.279998 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:26.279974    4421 scope.go:122] "RemoveContainer" containerID="299b38251681a7014b217ca8e00e3aac38bf96e66492f84cd2c6a9ad89ac04ab"
Nov 03 20:53:26.280333 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:53:26.280305    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"299b38251681a7014b217ca8e00e3aac38bf96e66492f84cd2c6a9ad89ac04ab\": container with ID starting with 299b38251681a7014b217ca8e00e3aac38bf96e66492f84cd2c6a9ad89ac04ab not found: ID does not exist" containerID="299b38251681a7014b217ca8e00e3aac38bf96e66492f84cd2c6a9ad89ac04ab"
Nov 03 20:53:26.280446 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:26.280341    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"299b38251681a7014b217ca8e00e3aac38bf96e66492f84cd2c6a9ad89ac04ab"} err="failed to get container status \"299b38251681a7014b217ca8e00e3aac38bf96e66492f84cd2c6a9ad89ac04ab\": rpc error: code = NotFound desc = could not find container \"299b38251681a7014b217ca8e00e3aac38bf96e66492f84cd2c6a9ad89ac04ab\": container with ID starting with 299b38251681a7014b217ca8e00e3aac38bf96e66492f84cd2c6a9ad89ac04ab not found: ID does not exist"
Nov 03 20:53:26.290520 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:26.290489    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"]
Nov 03 20:53:26.295200 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:26.295176    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kpfn4"]
Nov 03 20:53:28.019814 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:28.019778    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29af63ab-34a3-4ee3-9a74-b8108503642e" path="/var/lib/kubelet/pods/29af63ab-34a3-4ee3-9a74-b8108503642e/volumes"
Nov 03 20:53:28.314958 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:28.314878    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"]
Nov 03 20:53:28.315108 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:28.315064    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng" podUID="9f633998-421c-4d6f-9528-94114a4ca6fe" containerName="istio-proxy" containerID="cri-o://199e75df454098a534525ca8c8713ee9e2976670d496c84d58a476a4241fe7b6" gracePeriod=30
Nov 03 20:53:32.889061 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:32.889027    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:53:32.889452 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:32.889111    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng" podUID="9f633998-421c-4d6f-9528-94114a4ca6fe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:53:33.657539 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.657498    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:53:33.754976 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.754944    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-envoy\") pod \"9f633998-421c-4d6f-9528-94114a4ca6fe\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") "
Nov 03 20:53:33.755253 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.755019    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-workload-certs\") pod \"9f633998-421c-4d6f-9528-94114a4ca6fe\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") "
Nov 03 20:53:33.755253 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.755059    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f633998-421c-4d6f-9528-94114a4ca6fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f633998-421c-4d6f-9528-94114a4ca6fe-istiod-ca-cert\") pod \"9f633998-421c-4d6f-9528-94114a4ca6fe\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") "
Nov 03 20:53:33.755253 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.755104    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-workload-socket\") pod \"9f633998-421c-4d6f-9528-94114a4ca6fe\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") "
Nov 03 20:53:33.755253 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.755135    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-token\" (UniqueName: \"kubernetes.io/projected/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-token\") pod \"9f633998-421c-4d6f-9528-94114a4ca6fe\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") "
Nov 03 20:53:33.755579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.755366    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-podinfo\") pod \"9f633998-421c-4d6f-9528-94114a4ca6fe\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") "
Nov 03 20:53:33.755579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.755429    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-credential-socket\") pod \"9f633998-421c-4d6f-9528-94114a4ca6fe\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") "
Nov 03 20:53:33.755579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.755574    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-data\") pod \"9f633998-421c-4d6f-9528-94114a4ca6fe\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") "
Nov 03 20:53:33.755780 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.755703    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f633998-421c-4d6f-9528-94114a4ca6fe-kube-api-access-mdhds\" (UniqueName: \"kubernetes.io/projected/9f633998-421c-4d6f-9528-94114a4ca6fe-kube-api-access-mdhds\") pod \"9f633998-421c-4d6f-9528-94114a4ca6fe\" (UID: \"9f633998-421c-4d6f-9528-94114a4ca6fe\") "
Nov 03 20:53:33.755780 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.755716    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f633998-421c-4d6f-9528-94114a4ca6fe-istiod-ca-cert" pod "9f633998-421c-4d6f-9528-94114a4ca6fe" (UID: "9f633998-421c-4d6f-9528-94114a4ca6fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:53:33.756066 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.756040    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-credential-socket" pod "9f633998-421c-4d6f-9528-94114a4ca6fe" (UID: "9f633998-421c-4d6f-9528-94114a4ca6fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:53:33.756066 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.756047    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-workload-socket" pod "9f633998-421c-4d6f-9528-94114a4ca6fe" (UID: "9f633998-421c-4d6f-9528-94114a4ca6fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:53:33.756283 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.756255    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-workload-certs" pod "9f633998-421c-4d6f-9528-94114a4ca6fe" (UID: "9f633998-421c-4d6f-9528-94114a4ca6fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:53:33.756283 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.756271    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-data" pod "9f633998-421c-4d6f-9528-94114a4ca6fe" (UID: "9f633998-421c-4d6f-9528-94114a4ca6fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:53:33.756855 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.756829    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:33.756946 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.756861    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:33.756946 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.756876    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:33.756946 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.756891    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f633998-421c-4d6f-9528-94114a4ca6fe-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:33.756946 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.756907    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:33.758946 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.758921    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f633998-421c-4d6f-9528-94114a4ca6fe-kube-api-access-mdhds" pod "9f633998-421c-4d6f-9528-94114a4ca6fe" (UID: "9f633998-421c-4d6f-9528-94114a4ca6fe"). InnerVolumeSpecName "kube-api-access-mdhds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:53:33.758946 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.758932    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-envoy" pod "9f633998-421c-4d6f-9528-94114a4ca6fe" (UID: "9f633998-421c-4d6f-9528-94114a4ca6fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:53:33.758946 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.758940    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-podinfo" pod "9f633998-421c-4d6f-9528-94114a4ca6fe" (UID: "9f633998-421c-4d6f-9528-94114a4ca6fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:53:33.759403 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.759379    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-token" pod "9f633998-421c-4d6f-9528-94114a4ca6fe" (UID: "9f633998-421c-4d6f-9528-94114a4ca6fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:53:33.858560 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.858477    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:33.858560 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.858506    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdhds\" (UniqueName: \"kubernetes.io/projected/9f633998-421c-4d6f-9528-94114a4ca6fe-kube-api-access-mdhds\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:33.858560 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.858517    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:33.858560 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:33.858527    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f633998-421c-4d6f-9528-94114a4ca6fe-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:34.336777 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:34.336738    4421 generic.go:356] "Generic (PLEG): container finished" podID="9f633998-421c-4d6f-9528-94114a4ca6fe" containerID="199e75df454098a534525ca8c8713ee9e2976670d496c84d58a476a4241fe7b6" exitCode=0
Nov 03 20:53:34.337277 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:34.336793    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"
Nov 03 20:53:34.337277 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:34.336833    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng" event={"ID":"9f633998-421c-4d6f-9528-94114a4ca6fe","Type":"ContainerDied","Data":"199e75df454098a534525ca8c8713ee9e2976670d496c84d58a476a4241fe7b6"}
Nov 03 20:53:34.337277 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:34.336904    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng" event={"ID":"9f633998-421c-4d6f-9528-94114a4ca6fe","Type":"ContainerDied","Data":"4507924bdde3ccb95219bab3364df9c59617aa4d520a08fc71d656b6dfbbd560"}
Nov 03 20:53:34.337277 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:34.336938    4421 scope.go:122] "RemoveContainer" containerID="199e75df454098a534525ca8c8713ee9e2976670d496c84d58a476a4241fe7b6"
Nov 03 20:53:34.350251 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:34.350232    4421 scope.go:122] "RemoveContainer" containerID="199e75df454098a534525ca8c8713ee9e2976670d496c84d58a476a4241fe7b6"
Nov 03 20:53:34.350564 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:53:34.350543    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"199e75df454098a534525ca8c8713ee9e2976670d496c84d58a476a4241fe7b6\": container with ID starting with 199e75df454098a534525ca8c8713ee9e2976670d496c84d58a476a4241fe7b6 not found: ID does not exist" containerID="199e75df454098a534525ca8c8713ee9e2976670d496c84d58a476a4241fe7b6"
Nov 03 20:53:34.350721 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:34.350570    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"199e75df454098a534525ca8c8713ee9e2976670d496c84d58a476a4241fe7b6"} err="failed to get container status \"199e75df454098a534525ca8c8713ee9e2976670d496c84d58a476a4241fe7b6\": rpc error: code = NotFound desc = could not find container \"199e75df454098a534525ca8c8713ee9e2976670d496c84d58a476a4241fe7b6\": container with ID starting with 199e75df454098a534525ca8c8713ee9e2976670d496c84d58a476a4241fe7b6 not found: ID does not exist"
Nov 03 20:53:34.359839 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:34.359817    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"]
Nov 03 20:53:34.366957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:34.366934    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-fdpng"]
Nov 03 20:53:36.018833 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:36.018801    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f633998-421c-4d6f-9528-94114a4ca6fe" path="/var/lib/kubelet/pods/9f633998-421c-4d6f-9528-94114a4ca6fe/volumes"
Nov 03 20:53:49.888623 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:53:49.866799    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:53:53.766928 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:53.766884    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj"]
Nov 03 20:53:53.778948 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:53.778920    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr"]
Nov 03 20:53:58.767736 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:58.767696    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj" podUID="3a1011b5-fc48-46d6-845c-556012597a9c" containerName="payload-processing" containerID="cri-o://3bef8f5f405d70b13447b42c8c243a3a0fad7c1a3467c7a65dd0e9d9d47dd308" gracePeriod=25
Nov 03 20:53:58.780201 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:58.780170    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr" podUID="cf223c45-abe7-4e8c-88a3-b41ae1f1f25c" containerName="payload-pre-processing" containerID="cri-o://72f134bb42a0ec9dd21e225a2b55f0f3d9045374d76980a9a8dcb1f7f5f23635" gracePeriod=25
Nov 03 20:53:59.060100 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:59.060075    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj"
Nov 03 20:53:59.142950 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:59.142920    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a1011b5-fc48-46d6-845c-556012597a9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a1011b5-fc48-46d6-845c-556012597a9c-plugins-config-volume\") pod \"3a1011b5-fc48-46d6-845c-556012597a9c\" (UID: \"3a1011b5-fc48-46d6-845c-556012597a9c\") "
Nov 03 20:53:59.143166 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:59.143153    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a1011b5-fc48-46d6-845c-556012597a9c-kube-api-access-7r8k8\" (UniqueName: \"kubernetes.io/projected/3a1011b5-fc48-46d6-845c-556012597a9c-kube-api-access-7r8k8\") pod \"3a1011b5-fc48-46d6-845c-556012597a9c\" (UID: \"3a1011b5-fc48-46d6-845c-556012597a9c\") "
Nov 03 20:53:59.143380 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:59.143357    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a1011b5-fc48-46d6-845c-556012597a9c-plugins-config-volume" pod "3a1011b5-fc48-46d6-845c-556012597a9c" (UID: "3a1011b5-fc48-46d6-845c-556012597a9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:53:59.145131 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:59.145105    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a1011b5-fc48-46d6-845c-556012597a9c-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:59.146032 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:59.146009    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1011b5-fc48-46d6-845c-556012597a9c-kube-api-access-7r8k8" pod "3a1011b5-fc48-46d6-845c-556012597a9c" (UID: "3a1011b5-fc48-46d6-845c-556012597a9c"). InnerVolumeSpecName "kube-api-access-7r8k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:53:59.246649 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:59.246616    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r8k8\" (UniqueName: \"kubernetes.io/projected/3a1011b5-fc48-46d6-845c-556012597a9c-kube-api-access-7r8k8\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:59.677095 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:59.677061    4421 generic.go:356] "Generic (PLEG): container finished" podID="3a1011b5-fc48-46d6-845c-556012597a9c" containerID="3bef8f5f405d70b13447b42c8c243a3a0fad7c1a3467c7a65dd0e9d9d47dd308" exitCode=0
Nov 03 20:53:59.677279 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:59.677121    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj"
Nov 03 20:53:59.677279 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:59.677191    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj" event={"ID":"3a1011b5-fc48-46d6-845c-556012597a9c","Type":"ContainerDied","Data":"3bef8f5f405d70b13447b42c8c243a3a0fad7c1a3467c7a65dd0e9d9d47dd308"}
Nov 03 20:53:59.677279 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:59.677245    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj" event={"ID":"3a1011b5-fc48-46d6-845c-556012597a9c","Type":"ContainerDied","Data":"771cb841c0831c15c3606651c146cdba07cc21af09f494e7d5c38bbfd40a125d"}
Nov 03 20:53:59.677279 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:59.677267    4421 scope.go:122] "RemoveContainer" containerID="3bef8f5f405d70b13447b42c8c243a3a0fad7c1a3467c7a65dd0e9d9d47dd308"
Nov 03 20:53:59.690542 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:59.690523    4421 scope.go:122] "RemoveContainer" containerID="3bef8f5f405d70b13447b42c8c243a3a0fad7c1a3467c7a65dd0e9d9d47dd308"
Nov 03 20:53:59.690834 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:53:59.690817    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bef8f5f405d70b13447b42c8c243a3a0fad7c1a3467c7a65dd0e9d9d47dd308\": container with ID starting with 3bef8f5f405d70b13447b42c8c243a3a0fad7c1a3467c7a65dd0e9d9d47dd308 not found: ID does not exist" containerID="3bef8f5f405d70b13447b42c8c243a3a0fad7c1a3467c7a65dd0e9d9d47dd308"
Nov 03 20:53:59.690909 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:59.690840    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bef8f5f405d70b13447b42c8c243a3a0fad7c1a3467c7a65dd0e9d9d47dd308"} err="failed to get container status \"3bef8f5f405d70b13447b42c8c243a3a0fad7c1a3467c7a65dd0e9d9d47dd308\": rpc error: code = NotFound desc = could not find container \"3bef8f5f405d70b13447b42c8c243a3a0fad7c1a3467c7a65dd0e9d9d47dd308\": container with ID starting with 3bef8f5f405d70b13447b42c8c243a3a0fad7c1a3467c7a65dd0e9d9d47dd308 not found: ID does not exist"
Nov 03 20:53:59.703274 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:59.703254    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj"]
Nov 03 20:53:59.707830 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:53:59.707807    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bf7965-548c4c7466-mmstj"]
Nov 03 20:54:00.019438 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:00.019404    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a1011b5-fc48-46d6-845c-556012597a9c" path="/var/lib/kubelet/pods/3a1011b5-fc48-46d6-845c-556012597a9c/volumes"
Nov 03 20:54:03.488690 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:03.488651    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr" podUID="cf223c45-abe7-4e8c-88a3-b41ae1f1f25c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:54:13.489130 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:13.489091    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr" podUID="cf223c45-abe7-4e8c-88a3-b41ae1f1f25c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:54:19.073278 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:19.073251    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr"
Nov 03 20:54:19.131296 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:19.131193    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf223c45-abe7-4e8c-88a3-b41ae1f1f25c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf223c45-abe7-4e8c-88a3-b41ae1f1f25c-plugins-config-volume\") pod \"cf223c45-abe7-4e8c-88a3-b41ae1f1f25c\" (UID: \"cf223c45-abe7-4e8c-88a3-b41ae1f1f25c\") "
Nov 03 20:54:19.131669 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:19.131486    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf223c45-abe7-4e8c-88a3-b41ae1f1f25c-kube-api-access-vm866\" (UniqueName: \"kubernetes.io/projected/cf223c45-abe7-4e8c-88a3-b41ae1f1f25c-kube-api-access-vm866\") pod \"cf223c45-abe7-4e8c-88a3-b41ae1f1f25c\" (UID: \"cf223c45-abe7-4e8c-88a3-b41ae1f1f25c\") "
Nov 03 20:54:19.132587 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:19.132527    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf223c45-abe7-4e8c-88a3-b41ae1f1f25c-plugins-config-volume" pod "cf223c45-abe7-4e8c-88a3-b41ae1f1f25c" (UID: "cf223c45-abe7-4e8c-88a3-b41ae1f1f25c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:54:19.134231 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:19.134202    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf223c45-abe7-4e8c-88a3-b41ae1f1f25c-kube-api-access-vm866" pod "cf223c45-abe7-4e8c-88a3-b41ae1f1f25c" (UID: "cf223c45-abe7-4e8c-88a3-b41ae1f1f25c"). InnerVolumeSpecName "kube-api-access-vm866". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:54:19.233077 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:19.233036    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm866\" (UniqueName: \"kubernetes.io/projected/cf223c45-abe7-4e8c-88a3-b41ae1f1f25c-kube-api-access-vm866\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:54:19.233077 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:19.233069    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf223c45-abe7-4e8c-88a3-b41ae1f1f25c-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:54:19.630681 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:54:19.630647    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:54:19.876748 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:19.876712    4421 generic.go:356] "Generic (PLEG): container finished" podID="cf223c45-abe7-4e8c-88a3-b41ae1f1f25c" containerID="72f134bb42a0ec9dd21e225a2b55f0f3d9045374d76980a9a8dcb1f7f5f23635" exitCode=0
Nov 03 20:54:19.876922 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:19.876759    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr"
Nov 03 20:54:19.876922 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:19.876791    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr" event={"ID":"cf223c45-abe7-4e8c-88a3-b41ae1f1f25c","Type":"ContainerDied","Data":"72f134bb42a0ec9dd21e225a2b55f0f3d9045374d76980a9a8dcb1f7f5f23635"}
Nov 03 20:54:19.876922 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:19.876847    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr" event={"ID":"cf223c45-abe7-4e8c-88a3-b41ae1f1f25c","Type":"ContainerDied","Data":"27d061510abbf56cc3bc31452c3a6c63685bd088be8e20cd7d149b090e08e0db"}
Nov 03 20:54:19.876922 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:19.876872    4421 scope.go:122] "RemoveContainer" containerID="72f134bb42a0ec9dd21e225a2b55f0f3d9045374d76980a9a8dcb1f7f5f23635"
Nov 03 20:54:19.888358 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:19.888339    4421 scope.go:122] "RemoveContainer" containerID="72f134bb42a0ec9dd21e225a2b55f0f3d9045374d76980a9a8dcb1f7f5f23635"
Nov 03 20:54:19.888736 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:54:19.888716    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72f134bb42a0ec9dd21e225a2b55f0f3d9045374d76980a9a8dcb1f7f5f23635\": container with ID starting with 72f134bb42a0ec9dd21e225a2b55f0f3d9045374d76980a9a8dcb1f7f5f23635 not found: ID does not exist" containerID="72f134bb42a0ec9dd21e225a2b55f0f3d9045374d76980a9a8dcb1f7f5f23635"
Nov 03 20:54:19.888827 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:19.888744    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f134bb42a0ec9dd21e225a2b55f0f3d9045374d76980a9a8dcb1f7f5f23635"} err="failed to get container status \"72f134bb42a0ec9dd21e225a2b55f0f3d9045374d76980a9a8dcb1f7f5f23635\": rpc error: code = NotFound desc = could not find container \"72f134bb42a0ec9dd21e225a2b55f0f3d9045374d76980a9a8dcb1f7f5f23635\": container with ID starting with 72f134bb42a0ec9dd21e225a2b55f0f3d9045374d76980a9a8dcb1f7f5f23635 not found: ID does not exist"
Nov 03 20:54:19.904442 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:19.904406    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr"]
Nov 03 20:54:19.911117 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:19.911092    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-bf7965-55f8f68699-w4gkr"]
Nov 03 20:54:20.018471 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:20.018440    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf223c45-abe7-4e8c-88a3-b41ae1f1f25c" path="/var/lib/kubelet/pods/cf223c45-abe7-4e8c-88a3-b41ae1f1f25c/volumes"
Nov 03 20:54:49.631313 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:54:49.631264    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:54:49.918159 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:49.918071    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"]
Nov 03 20:54:49.919451 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:49.919423    4421 cpu_manager.go:414] "containerMap: removing container" podUID="9f633998-421c-4d6f-9528-94114a4ca6fe" containerName="istio-proxy"
Nov 03 20:54:49.919451 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:49.919452    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f633998-421c-4d6f-9528-94114a4ca6fe" containerName="istio-proxy"
Nov 03 20:54:49.919668 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:49.919498    4421 cpu_manager.go:414] "containerMap: removing container" podUID="cf223c45-abe7-4e8c-88a3-b41ae1f1f25c" containerName="payload-pre-processing"
Nov 03 20:54:49.919668 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:49.919507    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf223c45-abe7-4e8c-88a3-b41ae1f1f25c" containerName="payload-pre-processing"
Nov 03 20:54:49.919668 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:49.919561    4421 cpu_manager.go:414] "containerMap: removing container" podUID="3a1011b5-fc48-46d6-845c-556012597a9c" containerName="payload-processing"
Nov 03 20:54:49.919668 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:49.919569    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1011b5-fc48-46d6-845c-556012597a9c" containerName="payload-processing"
Nov 03 20:54:49.919668 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:49.919590    4421 cpu_manager.go:414] "containerMap: removing container" podUID="29af63ab-34a3-4ee3-9a74-b8108503642e" containerName="istio-proxy"
Nov 03 20:54:49.919668 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:49.919617    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29af63ab-34a3-4ee3-9a74-b8108503642e" containerName="istio-proxy"
Nov 03 20:54:49.920022 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:49.919848    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="29af63ab-34a3-4ee3-9a74-b8108503642e" containerName="istio-proxy"
Nov 03 20:54:49.920022 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:49.919873    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f633998-421c-4d6f-9528-94114a4ca6fe" containerName="istio-proxy"
Nov 03 20:54:49.920022 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:49.919888    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf223c45-abe7-4e8c-88a3-b41ae1f1f25c" containerName="payload-pre-processing"
Nov 03 20:54:49.920022 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:49.919905    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a1011b5-fc48-46d6-845c-556012597a9c" containerName="payload-processing"
Nov 03 20:54:49.926195 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:49.926176    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:49.936380 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:49.936253    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"]
Nov 03 20:54:49.949262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:49.949231    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f3a4c29a-openshift-default-dockercfg-h5jvk\""
Nov 03 20:54:50.056626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.054004    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-workload-certs\") pod \"e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") " pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.056626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.054074    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-credential-socket\") pod \"e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") " pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.056626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.054142    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-data\") pod \"e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") " pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.056626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.054175    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27g4d\" (UniqueName: \"kubernetes.io/projected/1106b419-0b67-47af-82eb-b2dd73d03f00-kube-api-access-27g4d\") pod \"e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") " pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.056626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.054237    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1106b419-0b67-47af-82eb-b2dd73d03f00-istiod-ca-cert\") pod \"e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") " pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.056626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.054333    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-podinfo\") pod \"e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") " pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.056626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.054443    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-workload-socket\") pod \"e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") " pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.056626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.054588    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-envoy\") pod \"e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") " pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.056626 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.054737    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-token\") pod \"e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") " pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.171682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.167957    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-envoy\") pod \"e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") " pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.171682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.168078    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-token\") pod \"e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") " pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.171682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.168143    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-workload-certs\") pod \"e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") " pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.171682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.168184    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-credential-socket\") pod \"e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") " pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.171682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.168268    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-data\") pod \"e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") " pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.171682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.168299    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27g4d\" (UniqueName: \"kubernetes.io/projected/1106b419-0b67-47af-82eb-b2dd73d03f00-kube-api-access-27g4d\") pod \"e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") " pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.171682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.168352    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1106b419-0b67-47af-82eb-b2dd73d03f00-istiod-ca-cert\") pod \"e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") " pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.171682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.168450    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-podinfo\") pod \"e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") " pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.171682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.168563    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-workload-socket\") pod \"e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") " pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.171682 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.169106    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-workload-socket\") pod \"e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") " pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.203676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.175138    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-data\") pod \"e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") " pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.203676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.175704    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-workload-certs\") pod \"e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") " pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.203676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.175936    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-credential-socket\") pod \"e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") " pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.203676 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.192077    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1106b419-0b67-47af-82eb-b2dd73d03f00-istiod-ca-cert\") pod \"e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") " pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.213219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.208412    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-envoy\") pod \"e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") " pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.213219 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.209529    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27g4d\" (UniqueName: \"kubernetes.io/projected/1106b419-0b67-47af-82eb-b2dd73d03f00-kube-api-access-27g4d\") pod \"e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") " pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.250480 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.244231    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-token\") pod \"e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") " pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.250480 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.245154    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-podinfo\") pod \"e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") " pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.250480 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.249893    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:50.852348 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.852311    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"]
Nov 03 20:54:50.852916 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.852866    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:54:50.853035 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.853013    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:54:50.853129 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:50.853107    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:54:51.171190 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:51.171089    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg" event={"ID":"1106b419-0b67-47af-82eb-b2dd73d03f00","Type":"ContainerStarted","Data":"abe80340a55eb2c611ebfe12bd352e7a85f0fb37a8e713be1b8006e4f9a336ec"}
Nov 03 20:54:51.171190 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:51.171162    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg" event={"ID":"1106b419-0b67-47af-82eb-b2dd73d03f00","Type":"ContainerStarted","Data":"e0214d7c6a952a35ff07cf4cf7774e3186a4d401711d1601ab4d6e1c0d3f3136"}
Nov 03 20:54:51.199334 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:51.199267    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg" podStartSLOduration=2.199245402 podStartE2EDuration="2.199245402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:51.194881805 +0000 UTC m=+3221.328010267" watchObservedRunningTime="2026-11-03 20:54:51.199245402 +0000 UTC m=+3221.332373865"
Nov 03 20:54:51.251923 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:51.251890    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:51.253503 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:51.253474    4421 patch_prober.go:29] interesting pod/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Nov 03 20:54:51.253661 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:51.253531    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg" podUID="1106b419-0b67-47af-82eb-b2dd73d03f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Nov 03 20:54:52.258144 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:52.258109    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:53.190090 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:53.190061    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:54:53.191204 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:54:53.191177    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:55:08.199684 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:08.198868    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98"]
Nov 03 20:55:08.206677 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:08.206647    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh"]
Nov 03 20:55:13.200098 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:13.200058    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98" podUID="6e96d27c-9c01-4e0b-b036-626ad4c4c08b" containerName="payload-processing" containerID="cri-o://020d55323d343e6ca670d5d08e0bf6a179e1198de1fe835e5ec021b613d830db" gracePeriod=25
Nov 03 20:55:13.207863 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:13.207822    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh" podUID="ca13e385-d5ab-41fe-87ce-7d903a01ee43" containerName="payload-pre-processing" containerID="cri-o://df0606a40e66596682cb92fed2c0edd589465e340573db647c5d956db7c4d5cb" gracePeriod=25
Nov 03 20:55:13.459572 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:13.459533    4421 generic.go:356] "Generic (PLEG): container finished" podID="6e96d27c-9c01-4e0b-b036-626ad4c4c08b" containerID="020d55323d343e6ca670d5d08e0bf6a179e1198de1fe835e5ec021b613d830db" exitCode=0
Nov 03 20:55:13.459902 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:13.459880    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98" event={"ID":"6e96d27c-9c01-4e0b-b036-626ad4c4c08b","Type":"ContainerDied","Data":"020d55323d343e6ca670d5d08e0bf6a179e1198de1fe835e5ec021b613d830db"}
Nov 03 20:55:13.504519 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:13.504488    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98"
Nov 03 20:55:13.683847 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:13.683764    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e96d27c-9c01-4e0b-b036-626ad4c4c08b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e96d27c-9c01-4e0b-b036-626ad4c4c08b-plugins-config-volume\") pod \"6e96d27c-9c01-4e0b-b036-626ad4c4c08b\" (UID: \"6e96d27c-9c01-4e0b-b036-626ad4c4c08b\") "
Nov 03 20:55:13.684026 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:13.683978    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e96d27c-9c01-4e0b-b036-626ad4c4c08b-kube-api-access-wflg4\" (UniqueName: \"kubernetes.io/projected/6e96d27c-9c01-4e0b-b036-626ad4c4c08b-kube-api-access-wflg4\") pod \"6e96d27c-9c01-4e0b-b036-626ad4c4c08b\" (UID: \"6e96d27c-9c01-4e0b-b036-626ad4c4c08b\") "
Nov 03 20:55:13.684181 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:13.684156    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e96d27c-9c01-4e0b-b036-626ad4c4c08b-plugins-config-volume" pod "6e96d27c-9c01-4e0b-b036-626ad4c4c08b" (UID: "6e96d27c-9c01-4e0b-b036-626ad4c4c08b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:55:13.684578 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:13.684556    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e96d27c-9c01-4e0b-b036-626ad4c4c08b-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:13.686197 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:13.686173    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e96d27c-9c01-4e0b-b036-626ad4c4c08b-kube-api-access-wflg4" pod "6e96d27c-9c01-4e0b-b036-626ad4c4c08b" (UID: "6e96d27c-9c01-4e0b-b036-626ad4c4c08b"). InnerVolumeSpecName "kube-api-access-wflg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:55:13.785994 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:13.785963    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wflg4\" (UniqueName: \"kubernetes.io/projected/6e96d27c-9c01-4e0b-b036-626ad4c4c08b-kube-api-access-wflg4\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:14.470093 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:14.470070    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98"
Nov 03 20:55:14.470517 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:14.470069    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98" event={"ID":"6e96d27c-9c01-4e0b-b036-626ad4c4c08b","Type":"ContainerDied","Data":"486627c791c20895e00e924ab77c5521d3ebfb0316d53cdfa62f7fab8840f398"}
Nov 03 20:55:14.470517 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:14.470196    4421 scope.go:122] "RemoveContainer" containerID="020d55323d343e6ca670d5d08e0bf6a179e1198de1fe835e5ec021b613d830db"
Nov 03 20:55:14.490278 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:14.490251    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98"]
Nov 03 20:55:14.494874 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:14.494852    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4d63cd-6f5ccff788-zcr98"]
Nov 03 20:55:16.025228 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:16.025029    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e96d27c-9c01-4e0b-b036-626ad4c4c08b" path="/var/lib/kubelet/pods/6e96d27c-9c01-4e0b-b036-626ad4c4c08b/volumes"
Nov 03 20:55:19.630567 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:55:19.630527    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:55:20.379338 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:20.379297    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh" podUID="ca13e385-d5ab-41fe-87ce-7d903a01ee43" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:55:22.726470 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:22.726433    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"]
Nov 03 20:55:22.727018 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:22.726749    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x" podUID="153d1bbf-3857-43c6-b66e-6dca99b48073" containerName="istio-proxy" containerID="cri-o://1b883b2063c3ada7a3c27f955ef31bd585a60455af4902149a8dff9b9bae067c" gracePeriod=30
Nov 03 20:55:26.304649 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:26.304616    4421 patch_prober.go:29] interesting pod/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:55:26.305062 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:26.304667    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x" podUID="153d1bbf-3857-43c6-b66e-6dca99b48073" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:55:27.980129 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:27.980103    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:55:28.136734 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.136649    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-data\") pod \"153d1bbf-3857-43c6-b66e-6dca99b48073\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") "
Nov 03 20:55:28.136734 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.136722    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-token\" (UniqueName: \"kubernetes.io/projected/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-token\") pod \"153d1bbf-3857-43c6-b66e-6dca99b48073\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") "
Nov 03 20:55:28.136957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.136869    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/153d1bbf-3857-43c6-b66e-6dca99b48073-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153d1bbf-3857-43c6-b66e-6dca99b48073-istiod-ca-cert\") pod \"153d1bbf-3857-43c6-b66e-6dca99b48073\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") "
Nov 03 20:55:28.136957 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.136903    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-podinfo\") pod \"153d1bbf-3857-43c6-b66e-6dca99b48073\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") "
Nov 03 20:55:28.137079 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.136966    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-credential-socket\") pod \"153d1bbf-3857-43c6-b66e-6dca99b48073\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") "
Nov 03 20:55:28.137079 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.136996    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-data" pod "153d1bbf-3857-43c6-b66e-6dca99b48073" (UID: "153d1bbf-3857-43c6-b66e-6dca99b48073"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:55:28.137079 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.137051    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-workload-socket\") pod \"153d1bbf-3857-43c6-b66e-6dca99b48073\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") "
Nov 03 20:55:28.137248 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.137091    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-envoy\") pod \"153d1bbf-3857-43c6-b66e-6dca99b48073\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") "
Nov 03 20:55:28.137248 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.137158    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/153d1bbf-3857-43c6-b66e-6dca99b48073-kube-api-access-wkxfd\" (UniqueName: \"kubernetes.io/projected/153d1bbf-3857-43c6-b66e-6dca99b48073-kube-api-access-wkxfd\") pod \"153d1bbf-3857-43c6-b66e-6dca99b48073\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") "
Nov 03 20:55:28.137406 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.137296    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-credential-socket" pod "153d1bbf-3857-43c6-b66e-6dca99b48073" (UID: "153d1bbf-3857-43c6-b66e-6dca99b48073"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:55:28.137406 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.137356    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-workload-certs\") pod \"153d1bbf-3857-43c6-b66e-6dca99b48073\" (UID: \"153d1bbf-3857-43c6-b66e-6dca99b48073\") "
Nov 03 20:55:28.137531 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.137414    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-workload-socket" pod "153d1bbf-3857-43c6-b66e-6dca99b48073" (UID: "153d1bbf-3857-43c6-b66e-6dca99b48073"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:55:28.137739 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.137714    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-workload-certs" pod "153d1bbf-3857-43c6-b66e-6dca99b48073" (UID: "153d1bbf-3857-43c6-b66e-6dca99b48073"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:55:28.138324 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.138295    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:28.138428 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.138330    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:28.138428 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.138344    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:28.138428 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.138358    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:28.138632 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.138511    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/153d1bbf-3857-43c6-b66e-6dca99b48073-istiod-ca-cert" pod "153d1bbf-3857-43c6-b66e-6dca99b48073" (UID: "153d1bbf-3857-43c6-b66e-6dca99b48073"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:55:28.140283 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.140254    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153d1bbf-3857-43c6-b66e-6dca99b48073-kube-api-access-wkxfd" pod "153d1bbf-3857-43c6-b66e-6dca99b48073" (UID: "153d1bbf-3857-43c6-b66e-6dca99b48073"). InnerVolumeSpecName "kube-api-access-wkxfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:55:28.140870 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.140836    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-podinfo" pod "153d1bbf-3857-43c6-b66e-6dca99b48073" (UID: "153d1bbf-3857-43c6-b66e-6dca99b48073"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:55:28.140972 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.140934    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-envoy" pod "153d1bbf-3857-43c6-b66e-6dca99b48073" (UID: "153d1bbf-3857-43c6-b66e-6dca99b48073"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:55:28.140972 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.140951    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-token" pod "153d1bbf-3857-43c6-b66e-6dca99b48073" (UID: "153d1bbf-3857-43c6-b66e-6dca99b48073"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:55:28.240113 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.240081    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:28.240113 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.240110    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/153d1bbf-3857-43c6-b66e-6dca99b48073-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:28.240113 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.240121    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:28.240337 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.240129    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/153d1bbf-3857-43c6-b66e-6dca99b48073-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:28.240337 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.240139    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkxfd\" (UniqueName: \"kubernetes.io/projected/153d1bbf-3857-43c6-b66e-6dca99b48073-kube-api-access-wkxfd\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:28.639765 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.639728    4421 generic.go:356] "Generic (PLEG): container finished" podID="153d1bbf-3857-43c6-b66e-6dca99b48073" containerID="1b883b2063c3ada7a3c27f955ef31bd585a60455af4902149a8dff9b9bae067c" exitCode=0
Nov 03 20:55:28.639980 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.639804    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"
Nov 03 20:55:28.639980 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.639825    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x" event={"ID":"153d1bbf-3857-43c6-b66e-6dca99b48073","Type":"ContainerDied","Data":"1b883b2063c3ada7a3c27f955ef31bd585a60455af4902149a8dff9b9bae067c"}
Nov 03 20:55:28.639980 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.639874    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x" event={"ID":"153d1bbf-3857-43c6-b66e-6dca99b48073","Type":"ContainerDied","Data":"0c5c775bfa2ed98ca5313ad7c9021b92c6ea597bb3ca6e125cdc237075cef26e"}
Nov 03 20:55:28.639980 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.639901    4421 scope.go:122] "RemoveContainer" containerID="1b883b2063c3ada7a3c27f955ef31bd585a60455af4902149a8dff9b9bae067c"
Nov 03 20:55:28.657763 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.657742    4421 scope.go:122] "RemoveContainer" containerID="1b883b2063c3ada7a3c27f955ef31bd585a60455af4902149a8dff9b9bae067c"
Nov 03 20:55:28.658038 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:55:28.658017    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b883b2063c3ada7a3c27f955ef31bd585a60455af4902149a8dff9b9bae067c\": container with ID starting with 1b883b2063c3ada7a3c27f955ef31bd585a60455af4902149a8dff9b9bae067c not found: ID does not exist" containerID="1b883b2063c3ada7a3c27f955ef31bd585a60455af4902149a8dff9b9bae067c"
Nov 03 20:55:28.658090 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.658048    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b883b2063c3ada7a3c27f955ef31bd585a60455af4902149a8dff9b9bae067c"} err="failed to get container status \"1b883b2063c3ada7a3c27f955ef31bd585a60455af4902149a8dff9b9bae067c\": rpc error: code = NotFound desc = could not find container \"1b883b2063c3ada7a3c27f955ef31bd585a60455af4902149a8dff9b9bae067c\": container with ID starting with 1b883b2063c3ada7a3c27f955ef31bd585a60455af4902149a8dff9b9bae067c not found: ID does not exist"
Nov 03 20:55:28.671543 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.671521    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"]
Nov 03 20:55:28.677336 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:28.677316    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4d63cd-openshift-default-58dc4588dc-5zn4x"]
Nov 03 20:55:29.812497 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:29.812464    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v"]
Nov 03 20:55:30.021856 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:30.021821    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="153d1bbf-3857-43c6-b66e-6dca99b48073" path="/var/lib/kubelet/pods/153d1bbf-3857-43c6-b66e-6dca99b48073/volumes"
Nov 03 20:55:30.379567 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:30.379529    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh" podUID="ca13e385-d5ab-41fe-87ce-7d903a01ee43" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:55:33.490081 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:33.490053    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh"
Nov 03 20:55:33.657551 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:33.657509    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca13e385-d5ab-41fe-87ce-7d903a01ee43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca13e385-d5ab-41fe-87ce-7d903a01ee43-plugins-config-volume\") pod \"ca13e385-d5ab-41fe-87ce-7d903a01ee43\" (UID: \"ca13e385-d5ab-41fe-87ce-7d903a01ee43\") "
Nov 03 20:55:33.657831 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:33.657800    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca13e385-d5ab-41fe-87ce-7d903a01ee43-kube-api-access-pkswx\" (UniqueName: \"kubernetes.io/projected/ca13e385-d5ab-41fe-87ce-7d903a01ee43-kube-api-access-pkswx\") pod \"ca13e385-d5ab-41fe-87ce-7d903a01ee43\" (UID: \"ca13e385-d5ab-41fe-87ce-7d903a01ee43\") "
Nov 03 20:55:33.657969 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:33.657948    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca13e385-d5ab-41fe-87ce-7d903a01ee43-plugins-config-volume" pod "ca13e385-d5ab-41fe-87ce-7d903a01ee43" (UID: "ca13e385-d5ab-41fe-87ce-7d903a01ee43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:55:33.658489 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:33.658466    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca13e385-d5ab-41fe-87ce-7d903a01ee43-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:33.660254 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:33.660229    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca13e385-d5ab-41fe-87ce-7d903a01ee43-kube-api-access-pkswx" pod "ca13e385-d5ab-41fe-87ce-7d903a01ee43" (UID: "ca13e385-d5ab-41fe-87ce-7d903a01ee43"). InnerVolumeSpecName "kube-api-access-pkswx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:55:33.685822 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:33.685789    4421 generic.go:356] "Generic (PLEG): container finished" podID="ca13e385-d5ab-41fe-87ce-7d903a01ee43" containerID="df0606a40e66596682cb92fed2c0edd589465e340573db647c5d956db7c4d5cb" exitCode=0
Nov 03 20:55:33.685951 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:33.685847    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh"
Nov 03 20:55:33.685951 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:33.685871    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh" event={"ID":"ca13e385-d5ab-41fe-87ce-7d903a01ee43","Type":"ContainerDied","Data":"df0606a40e66596682cb92fed2c0edd589465e340573db647c5d956db7c4d5cb"}
Nov 03 20:55:33.685951 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:33.685941    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh" event={"ID":"ca13e385-d5ab-41fe-87ce-7d903a01ee43","Type":"ContainerDied","Data":"3da672e750beaf92790213244120cedf8097f234150e5c005747277b7e44ce5f"}
Nov 03 20:55:33.686123 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:33.685971    4421 scope.go:122] "RemoveContainer" containerID="df0606a40e66596682cb92fed2c0edd589465e340573db647c5d956db7c4d5cb"
Nov 03 20:55:33.698257 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:33.698240    4421 scope.go:122] "RemoveContainer" containerID="df0606a40e66596682cb92fed2c0edd589465e340573db647c5d956db7c4d5cb"
Nov 03 20:55:33.698546 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:55:33.698526    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df0606a40e66596682cb92fed2c0edd589465e340573db647c5d956db7c4d5cb\": container with ID starting with df0606a40e66596682cb92fed2c0edd589465e340573db647c5d956db7c4d5cb not found: ID does not exist" containerID="df0606a40e66596682cb92fed2c0edd589465e340573db647c5d956db7c4d5cb"
Nov 03 20:55:33.698627 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:33.698554    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df0606a40e66596682cb92fed2c0edd589465e340573db647c5d956db7c4d5cb"} err="failed to get container status \"df0606a40e66596682cb92fed2c0edd589465e340573db647c5d956db7c4d5cb\": rpc error: code = NotFound desc = could not find container \"df0606a40e66596682cb92fed2c0edd589465e340573db647c5d956db7c4d5cb\": container with ID starting with df0606a40e66596682cb92fed2c0edd589465e340573db647c5d956db7c4d5cb not found: ID does not exist"
Nov 03 20:55:33.710566 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:33.710546    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh"]
Nov 03 20:55:33.718756 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:33.718702    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4d63cd-5d7fdcb7cf-ktxzh"]
Nov 03 20:55:33.760635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:33.760238    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkswx\" (UniqueName: \"kubernetes.io/projected/ca13e385-d5ab-41fe-87ce-7d903a01ee43-kube-api-access-pkswx\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:34.018324 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:34.018247    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca13e385-d5ab-41fe-87ce-7d903a01ee43" path="/var/lib/kubelet/pods/ca13e385-d5ab-41fe-87ce-7d903a01ee43/volumes"
Nov 03 20:55:34.813614 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:34.813570    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v" podUID="f399b4d8-6505-4da5-a9d2-e772cc7071d2" containerName="payload-pre-processing" containerID="cri-o://e5e2f217f943d3c4a96d1dd84ad570f9ea62e6c7aff1da7119bafd98c1b3acf5" gracePeriod=25
Nov 03 20:55:35.091679 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:35.084798    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v" podUID="f399b4d8-6505-4da5-a9d2-e772cc7071d2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:55:45.081223 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:45.081183    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v" podUID="f399b4d8-6505-4da5-a9d2-e772cc7071d2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:55:49.042120 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:49.042085    4421 generic.go:356] "Generic (PLEG): container finished" podID="1106b419-0b67-47af-82eb-b2dd73d03f00" containerID="abe80340a55eb2c611ebfe12bd352e7a85f0fb37a8e713be1b8006e4f9a336ec" exitCode=0
Nov 03 20:55:49.042704 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:49.042166    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg" event={"ID":"1106b419-0b67-47af-82eb-b2dd73d03f00","Type":"ContainerDied","Data":"abe80340a55eb2c611ebfe12bd352e7a85f0fb37a8e713be1b8006e4f9a336ec"}
Nov 03 20:55:49.042791 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:49.042772    4421 scope.go:122] "RemoveContainer" containerID="abe80340a55eb2c611ebfe12bd352e7a85f0fb37a8e713be1b8006e4f9a336ec"
Nov 03 20:55:49.045226 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:49.045191    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:55:49.045340 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:49.045286    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:55:49.045340 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:49.045329    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:55:49.630232 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:55:49.630195    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:55:50.086235 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:50.086198    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg" event={"ID":"1106b419-0b67-47af-82eb-b2dd73d03f00","Type":"ContainerStarted","Data":"a1994fe1b41c29d145b1822bf32d11cd21a1ff32e82a68caa13eabd51088449a"}
Nov 03 20:55:50.263623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:50.258825    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:55:50.263623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:50.261713    4421 patch_prober.go:29] interesting pod/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Nov 03 20:55:50.263623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:50.261781    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg" podUID="1106b419-0b67-47af-82eb-b2dd73d03f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Nov 03 20:55:51.257625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:51.255679    4421 patch_prober.go:29] interesting pod/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Nov 03 20:55:51.257625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:51.255743    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg" podUID="1106b419-0b67-47af-82eb-b2dd73d03f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Nov 03 20:55:52.264947 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:52.264909    4421 patch_prober.go:29] interesting pod/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Nov 03 20:55:52.265571 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:52.265540    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg" podUID="1106b419-0b67-47af-82eb-b2dd73d03f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Nov 03 20:55:53.190008 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:53.189976    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:55:53.263624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:53.260960    4421 patch_prober.go:29] interesting pod/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Nov 03 20:55:53.263624 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:53.261020    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg" podUID="1106b419-0b67-47af-82eb-b2dd73d03f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Nov 03 20:55:54.260656 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:54.255025    4421 patch_prober.go:29] interesting pod/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Nov 03 20:55:54.260656 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:54.255094    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg" podUID="1106b419-0b67-47af-82eb-b2dd73d03f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Nov 03 20:55:55.131611 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:55.131565    4421 generic.go:356] "Generic (PLEG): container finished" podID="f399b4d8-6505-4da5-a9d2-e772cc7071d2" containerID="e5e2f217f943d3c4a96d1dd84ad570f9ea62e6c7aff1da7119bafd98c1b3acf5" exitCode=0
Nov 03 20:55:55.131806 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:55.131655    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v" event={"ID":"f399b4d8-6505-4da5-a9d2-e772cc7071d2","Type":"ContainerDied","Data":"e5e2f217f943d3c4a96d1dd84ad570f9ea62e6c7aff1da7119bafd98c1b3acf5"}
Nov 03 20:55:55.180902 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:55.180878    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v"
Nov 03 20:55:55.260342 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:55.256890    4421 patch_prober.go:29] interesting pod/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Nov 03 20:55:55.260342 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:55.256949    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg" podUID="1106b419-0b67-47af-82eb-b2dd73d03f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Nov 03 20:55:55.275191 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:55.275079    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f399b4d8-6505-4da5-a9d2-e772cc7071d2-kube-api-access-wvb5x\" (UniqueName: \"kubernetes.io/projected/f399b4d8-6505-4da5-a9d2-e772cc7071d2-kube-api-access-wvb5x\") pod \"f399b4d8-6505-4da5-a9d2-e772cc7071d2\" (UID: \"f399b4d8-6505-4da5-a9d2-e772cc7071d2\") "
Nov 03 20:55:55.275733 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:55.275348    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f399b4d8-6505-4da5-a9d2-e772cc7071d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f399b4d8-6505-4da5-a9d2-e772cc7071d2-plugins-config-volume\") pod \"f399b4d8-6505-4da5-a9d2-e772cc7071d2\" (UID: \"f399b4d8-6505-4da5-a9d2-e772cc7071d2\") "
Nov 03 20:55:55.275733 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:55.275713    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f399b4d8-6505-4da5-a9d2-e772cc7071d2-plugins-config-volume" pod "f399b4d8-6505-4da5-a9d2-e772cc7071d2" (UID: "f399b4d8-6505-4da5-a9d2-e772cc7071d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:55:55.277030 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:55.276992    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f399b4d8-6505-4da5-a9d2-e772cc7071d2-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:55.278052 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:55.278020    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f399b4d8-6505-4da5-a9d2-e772cc7071d2-kube-api-access-wvb5x" pod "f399b4d8-6505-4da5-a9d2-e772cc7071d2" (UID: "f399b4d8-6505-4da5-a9d2-e772cc7071d2"). InnerVolumeSpecName "kube-api-access-wvb5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:55:55.395706 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:55.381644    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvb5x\" (UniqueName: \"kubernetes.io/projected/f399b4d8-6505-4da5-a9d2-e772cc7071d2-kube-api-access-wvb5x\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:56.083805 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:56.083756    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v" podUID="f399b4d8-6505-4da5-a9d2-e772cc7071d2" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.245:50052\" within 1s: context deadline exceeded"
Nov 03 20:55:56.084003 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:56.083843    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v"
Nov 03 20:55:56.084003 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:55:56.083863    4421 logging.go:55] [core] [Channel #15956 SubChannel #15957]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.245:50052", ServerName: "10.134.0.245:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.245:50052: operation was canceled"
Nov 03 20:55:56.161143 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:56.161109    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v" event={"ID":"f399b4d8-6505-4da5-a9d2-e772cc7071d2","Type":"ContainerDied","Data":"b09938c920aa37c7f86afa946f82022f1de47cd79e366417dd96c03195b38875"}
Nov 03 20:55:56.161143 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:56.161135    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v"
Nov 03 20:55:56.161417 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:56.161177    4421 scope.go:122] "RemoveContainer" containerID="e5e2f217f943d3c4a96d1dd84ad570f9ea62e6c7aff1da7119bafd98c1b3acf5"
Nov 03 20:55:56.188072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:56.188032    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v"]
Nov 03 20:55:56.198105 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:56.198053    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-519a78-6dc895887-8l54v"]
Nov 03 20:55:56.257282 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:56.257251    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:55:56.258582 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:56.258562    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:55:58.018979 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:55:58.018947    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f399b4d8-6505-4da5-a9d2-e772cc7071d2" path="/var/lib/kubelet/pods/f399b4d8-6505-4da5-a9d2-e772cc7071d2/volumes"
Nov 03 20:56:19.630714 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:56:19.630676    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:56:49.630489 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:56:49.630451    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:57:02.140612 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:02.140563    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"]
Nov 03 20:57:02.895526 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:02.895453    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn" podUID="1e5d4c6c-d6ce-4a10-b875-6016aaf24e65" containerName="istio-proxy" containerID="cri-o://dc022417d717572f02b905d78c3c77a7978d66cced992ce85aaddb5f1992d3a1" gracePeriod=30
Nov 03 20:57:07.743201 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:07.743155    4421 patch_prober.go:29] interesting pod/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:57:07.743778 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:07.743221    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn" podUID="1e5d4c6c-d6ce-4a10-b875-6016aaf24e65" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:57:08.178592 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.178559    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"]
Nov 03 20:57:08.178860 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.178835    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh" podUID="d5b6a788-2a4f-4819-8943-5fe88488888c" containerName="istio-proxy" containerID="cri-o://2410dac0bb78878e260f747d14b506782b5fcd409a9a7e426c670a22552926d0" gracePeriod=30
Nov 03 20:57:08.212216 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.212185    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:57:08.241916 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.241886    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-podinfo\") pod \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") "
Nov 03 20:57:08.242105 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.241927    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-workload-certs\") pod \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") "
Nov 03 20:57:08.242105 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.241954    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-envoy\") pod \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") "
Nov 03 20:57:08.242105 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.242055    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-workload-socket\") pod \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") "
Nov 03 20:57:08.242105 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.242093    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-kube-api-access-7kjb4\" (UniqueName: \"kubernetes.io/projected/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-kube-api-access-7kjb4\") pod \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") "
Nov 03 20:57:08.242336 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.242175    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istiod-ca-cert\") pod \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") "
Nov 03 20:57:08.242336 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.242216    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-workload-certs" pod "1e5d4c6c-d6ce-4a10-b875-6016aaf24e65" (UID: "1e5d4c6c-d6ce-4a10-b875-6016aaf24e65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:08.242336 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.242269    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-credential-socket\") pod \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") "
Nov 03 20:57:08.244414 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.242339    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-token\" (UniqueName: \"kubernetes.io/projected/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-token\") pod \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") "
Nov 03 20:57:08.244414 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.242667    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istiod-ca-cert" pod "1e5d4c6c-d6ce-4a10-b875-6016aaf24e65" (UID: "1e5d4c6c-d6ce-4a10-b875-6016aaf24e65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:57:08.244414 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.242853    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-workload-socket" pod "1e5d4c6c-d6ce-4a10-b875-6016aaf24e65" (UID: "1e5d4c6c-d6ce-4a10-b875-6016aaf24e65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:08.244414 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.244219    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-credential-socket" pod "1e5d4c6c-d6ce-4a10-b875-6016aaf24e65" (UID: "1e5d4c6c-d6ce-4a10-b875-6016aaf24e65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:08.245785 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.245760    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-data\") pod \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\" (UID: \"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65\") "
Nov 03 20:57:08.247048 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.247023    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:08.247152 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.247070    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:08.247152 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.247086    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:08.247152 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.247101    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:08.247364 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.247344    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-data" pod "1e5d4c6c-d6ce-4a10-b875-6016aaf24e65" (UID: "1e5d4c6c-d6ce-4a10-b875-6016aaf24e65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:08.247736 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.247710    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-token" pod "1e5d4c6c-d6ce-4a10-b875-6016aaf24e65" (UID: "1e5d4c6c-d6ce-4a10-b875-6016aaf24e65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:57:08.248722 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.248700    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-envoy" pod "1e5d4c6c-d6ce-4a10-b875-6016aaf24e65" (UID: "1e5d4c6c-d6ce-4a10-b875-6016aaf24e65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:08.251990 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.251962    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-podinfo" pod "1e5d4c6c-d6ce-4a10-b875-6016aaf24e65" (UID: "1e5d4c6c-d6ce-4a10-b875-6016aaf24e65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:57:08.253933 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.253903    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-kube-api-access-7kjb4" pod "1e5d4c6c-d6ce-4a10-b875-6016aaf24e65" (UID: "1e5d4c6c-d6ce-4a10-b875-6016aaf24e65"). InnerVolumeSpecName "kube-api-access-7kjb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:57:08.348332 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.348296    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:08.348332 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.348330    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:08.348541 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.348343    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:08.348541 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.348358    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kjb4\" (UniqueName: \"kubernetes.io/projected/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-kube-api-access-7kjb4\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:08.348541 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.348372    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:08.947670 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.947636    4421 generic.go:356] "Generic (PLEG): container finished" podID="1e5d4c6c-d6ce-4a10-b875-6016aaf24e65" containerID="dc022417d717572f02b905d78c3c77a7978d66cced992ce85aaddb5f1992d3a1" exitCode=0
Nov 03 20:57:08.948166 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.947712    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"
Nov 03 20:57:08.948166 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.947720    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn" event={"ID":"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65","Type":"ContainerDied","Data":"dc022417d717572f02b905d78c3c77a7978d66cced992ce85aaddb5f1992d3a1"}
Nov 03 20:57:08.948166 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.947804    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn" event={"ID":"1e5d4c6c-d6ce-4a10-b875-6016aaf24e65","Type":"ContainerDied","Data":"a0d7ea33af6e5d525eac87778417e0ac04afa59130a69f115374ecdd0befa439"}
Nov 03 20:57:08.948166 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.947856    4421 scope.go:122] "RemoveContainer" containerID="dc022417d717572f02b905d78c3c77a7978d66cced992ce85aaddb5f1992d3a1"
Nov 03 20:57:08.961161 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.961140    4421 scope.go:122] "RemoveContainer" containerID="dc022417d717572f02b905d78c3c77a7978d66cced992ce85aaddb5f1992d3a1"
Nov 03 20:57:08.961564 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:57:08.961534    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc022417d717572f02b905d78c3c77a7978d66cced992ce85aaddb5f1992d3a1\": container with ID starting with dc022417d717572f02b905d78c3c77a7978d66cced992ce85aaddb5f1992d3a1 not found: ID does not exist" containerID="dc022417d717572f02b905d78c3c77a7978d66cced992ce85aaddb5f1992d3a1"
Nov 03 20:57:08.961720 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.961570    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc022417d717572f02b905d78c3c77a7978d66cced992ce85aaddb5f1992d3a1"} err="failed to get container status \"dc022417d717572f02b905d78c3c77a7978d66cced992ce85aaddb5f1992d3a1\": rpc error: code = NotFound desc = could not find container \"dc022417d717572f02b905d78c3c77a7978d66cced992ce85aaddb5f1992d3a1\": container with ID starting with dc022417d717572f02b905d78c3c77a7978d66cced992ce85aaddb5f1992d3a1 not found: ID does not exist"
Nov 03 20:57:08.985380 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.985344    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"]
Nov 03 20:57:08.992444 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:08.992415    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-37aa95-openshift-default-75559b5549-h8nqn"]
Nov 03 20:57:10.021661 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:10.020689    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e5d4c6c-d6ce-4a10-b875-6016aaf24e65" path="/var/lib/kubelet/pods/1e5d4c6c-d6ce-4a10-b875-6016aaf24e65/volumes"
Nov 03 20:57:13.561309 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.561283    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:57:13.664084 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.664047    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-podinfo\") pod \"d5b6a788-2a4f-4819-8943-5fe88488888c\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") "
Nov 03 20:57:13.664243 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.664097    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-workload-certs\") pod \"d5b6a788-2a4f-4819-8943-5fe88488888c\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") "
Nov 03 20:57:13.664243 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.664149    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5b6a788-2a4f-4819-8943-5fe88488888c-kube-api-access-xtl8n\" (UniqueName: \"kubernetes.io/projected/d5b6a788-2a4f-4819-8943-5fe88488888c-kube-api-access-xtl8n\") pod \"d5b6a788-2a4f-4819-8943-5fe88488888c\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") "
Nov 03 20:57:13.664243 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.664176    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-envoy\") pod \"d5b6a788-2a4f-4819-8943-5fe88488888c\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") "
Nov 03 20:57:13.664243 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.664221    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-workload-socket\") pod \"d5b6a788-2a4f-4819-8943-5fe88488888c\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") "
Nov 03 20:57:13.664481 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.664248    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-token\" (UniqueName: \"kubernetes.io/projected/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-token\") pod \"d5b6a788-2a4f-4819-8943-5fe88488888c\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") "
Nov 03 20:57:13.664481 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.664322    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-credential-socket\") pod \"d5b6a788-2a4f-4819-8943-5fe88488888c\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") "
Nov 03 20:57:13.664481 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.664436    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5b6a788-2a4f-4819-8943-5fe88488888c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b6a788-2a4f-4819-8943-5fe88488888c-istiod-ca-cert\") pod \"d5b6a788-2a4f-4819-8943-5fe88488888c\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") "
Nov 03 20:57:13.664825 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.664494    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-workload-certs" pod "d5b6a788-2a4f-4819-8943-5fe88488888c" (UID: "d5b6a788-2a4f-4819-8943-5fe88488888c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:13.664825 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.664637    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-data\") pod \"d5b6a788-2a4f-4819-8943-5fe88488888c\" (UID: \"d5b6a788-2a4f-4819-8943-5fe88488888c\") "
Nov 03 20:57:13.664954 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.664922    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-credential-socket" pod "d5b6a788-2a4f-4819-8943-5fe88488888c" (UID: "d5b6a788-2a4f-4819-8943-5fe88488888c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:13.665010 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.664949    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b6a788-2a4f-4819-8943-5fe88488888c-istiod-ca-cert" pod "d5b6a788-2a4f-4819-8943-5fe88488888c" (UID: "d5b6a788-2a4f-4819-8943-5fe88488888c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:57:13.665134 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.665104    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-data" pod "d5b6a788-2a4f-4819-8943-5fe88488888c" (UID: "d5b6a788-2a4f-4819-8943-5fe88488888c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:13.665355 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.665301    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-workload-socket" pod "d5b6a788-2a4f-4819-8943-5fe88488888c" (UID: "d5b6a788-2a4f-4819-8943-5fe88488888c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:13.665635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.665587    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:13.665717 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.665641    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:13.665717 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.665656    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:13.665717 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.665670    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:13.665717 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.665684    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b6a788-2a4f-4819-8943-5fe88488888c-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:13.667339 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.667280    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-token" pod "d5b6a788-2a4f-4819-8943-5fe88488888c" (UID: "d5b6a788-2a4f-4819-8943-5fe88488888c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:57:13.667339 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.667317    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b6a788-2a4f-4819-8943-5fe88488888c-kube-api-access-xtl8n" pod "d5b6a788-2a4f-4819-8943-5fe88488888c" (UID: "d5b6a788-2a4f-4819-8943-5fe88488888c"). InnerVolumeSpecName "kube-api-access-xtl8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:57:13.667339 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.667317    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-podinfo" pod "d5b6a788-2a4f-4819-8943-5fe88488888c" (UID: "d5b6a788-2a4f-4819-8943-5fe88488888c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:57:13.667558 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.667524    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-envoy" pod "d5b6a788-2a4f-4819-8943-5fe88488888c" (UID: "d5b6a788-2a4f-4819-8943-5fe88488888c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:13.768060 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.768018    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:13.768060 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.768054    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtl8n\" (UniqueName: \"kubernetes.io/projected/d5b6a788-2a4f-4819-8943-5fe88488888c-kube-api-access-xtl8n\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:13.768060 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.768066    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:13.768378 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:13.768075    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5b6a788-2a4f-4819-8943-5fe88488888c-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:14.039936 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:14.039887    4421 generic.go:356] "Generic (PLEG): container finished" podID="d5b6a788-2a4f-4819-8943-5fe88488888c" containerID="2410dac0bb78878e260f747d14b506782b5fcd409a9a7e426c670a22552926d0" exitCode=0
Nov 03 20:57:14.040119 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:14.039943    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"
Nov 03 20:57:14.040119 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:14.039996    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh" event={"ID":"d5b6a788-2a4f-4819-8943-5fe88488888c","Type":"ContainerDied","Data":"2410dac0bb78878e260f747d14b506782b5fcd409a9a7e426c670a22552926d0"}
Nov 03 20:57:14.040119 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:14.040039    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh" event={"ID":"d5b6a788-2a4f-4819-8943-5fe88488888c","Type":"ContainerDied","Data":"d3a316fddcb39b4641241c14ede46cb19a589b144f49a527c2cc9802d0227ec2"}
Nov 03 20:57:14.040119 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:14.040069    4421 scope.go:122] "RemoveContainer" containerID="2410dac0bb78878e260f747d14b506782b5fcd409a9a7e426c670a22552926d0"
Nov 03 20:57:14.057740 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:14.057712    4421 scope.go:122] "RemoveContainer" containerID="2410dac0bb78878e260f747d14b506782b5fcd409a9a7e426c670a22552926d0"
Nov 03 20:57:14.060529 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:57:14.060456    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2410dac0bb78878e260f747d14b506782b5fcd409a9a7e426c670a22552926d0\": container with ID starting with 2410dac0bb78878e260f747d14b506782b5fcd409a9a7e426c670a22552926d0 not found: ID does not exist" containerID="2410dac0bb78878e260f747d14b506782b5fcd409a9a7e426c670a22552926d0"
Nov 03 20:57:14.060529 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:14.060499    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2410dac0bb78878e260f747d14b506782b5fcd409a9a7e426c670a22552926d0"} err="failed to get container status \"2410dac0bb78878e260f747d14b506782b5fcd409a9a7e426c670a22552926d0\": rpc error: code = NotFound desc = could not find container \"2410dac0bb78878e260f747d14b506782b5fcd409a9a7e426c670a22552926d0\": container with ID starting with 2410dac0bb78878e260f747d14b506782b5fcd409a9a7e426c670a22552926d0 not found: ID does not exist"
Nov 03 20:57:14.070668 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:14.070642    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"]
Nov 03 20:57:14.076112 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:14.076084    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-hwznh"]
Nov 03 20:57:16.017999 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:16.017965    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5b6a788-2a4f-4819-8943-5fe88488888c" path="/var/lib/kubelet/pods/d5b6a788-2a4f-4819-8943-5fe88488888c/volumes"
Nov 03 20:57:19.630282 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:57:19.630196    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:57:26.214084 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:26.214037    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"]
Nov 03 20:57:26.214912 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:26.214879    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg" podUID="1106b419-0b67-47af-82eb-b2dd73d03f00" containerName="istio-proxy" containerID="cri-o://a1994fe1b41c29d145b1822bf32d11cd21a1ff32e82a68caa13eabd51088449a" gracePeriod=30
Nov 03 20:57:26.260331 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:26.260293    4421 patch_prober.go:29] interesting pod/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:57:26.260514 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:26.260375    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg" podUID="1106b419-0b67-47af-82eb-b2dd73d03f00" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:57:31.518844 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.518810    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:57:31.580419 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.580377    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-podinfo\") pod \"1106b419-0b67-47af-82eb-b2dd73d03f00\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") "
Nov 03 20:57:31.580716 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.580473    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-workload-certs\") pod \"1106b419-0b67-47af-82eb-b2dd73d03f00\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") "
Nov 03 20:57:31.580716 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.580501    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-data\") pod \"1106b419-0b67-47af-82eb-b2dd73d03f00\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") "
Nov 03 20:57:31.580716 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.580528    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-envoy\") pod \"1106b419-0b67-47af-82eb-b2dd73d03f00\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") "
Nov 03 20:57:31.580981 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.580954    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-workload-socket\") pod \"1106b419-0b67-47af-82eb-b2dd73d03f00\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") "
Nov 03 20:57:31.581067 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.581028    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-token\" (UniqueName: \"kubernetes.io/projected/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-token\") pod \"1106b419-0b67-47af-82eb-b2dd73d03f00\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") "
Nov 03 20:57:31.581149 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.581135    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1106b419-0b67-47af-82eb-b2dd73d03f00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1106b419-0b67-47af-82eb-b2dd73d03f00-istiod-ca-cert\") pod \"1106b419-0b67-47af-82eb-b2dd73d03f00\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") "
Nov 03 20:57:31.581262 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.581244    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1106b419-0b67-47af-82eb-b2dd73d03f00-kube-api-access-27g4d\" (UniqueName: \"kubernetes.io/projected/1106b419-0b67-47af-82eb-b2dd73d03f00-kube-api-access-27g4d\") pod \"1106b419-0b67-47af-82eb-b2dd73d03f00\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") "
Nov 03 20:57:31.581376 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.581362    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-credential-socket\") pod \"1106b419-0b67-47af-82eb-b2dd73d03f00\" (UID: \"1106b419-0b67-47af-82eb-b2dd73d03f00\") "
Nov 03 20:57:31.581735 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.581704    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-workload-certs" pod "1106b419-0b67-47af-82eb-b2dd73d03f00" (UID: "1106b419-0b67-47af-82eb-b2dd73d03f00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:31.581986 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.581962    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-credential-socket" pod "1106b419-0b67-47af-82eb-b2dd73d03f00" (UID: "1106b419-0b67-47af-82eb-b2dd73d03f00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:31.582241 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.582221    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:31.582322 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.582244    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:31.582735 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.582675    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-data" pod "1106b419-0b67-47af-82eb-b2dd73d03f00" (UID: "1106b419-0b67-47af-82eb-b2dd73d03f00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:31.583235 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.583181    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-workload-socket" pod "1106b419-0b67-47af-82eb-b2dd73d03f00" (UID: "1106b419-0b67-47af-82eb-b2dd73d03f00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:31.583563 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.583524    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1106b419-0b67-47af-82eb-b2dd73d03f00-istiod-ca-cert" pod "1106b419-0b67-47af-82eb-b2dd73d03f00" (UID: "1106b419-0b67-47af-82eb-b2dd73d03f00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:57:31.585536 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.584973    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1106b419-0b67-47af-82eb-b2dd73d03f00-kube-api-access-27g4d" pod "1106b419-0b67-47af-82eb-b2dd73d03f00" (UID: "1106b419-0b67-47af-82eb-b2dd73d03f00"). InnerVolumeSpecName "kube-api-access-27g4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:57:31.586936 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.586916    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-envoy" pod "1106b419-0b67-47af-82eb-b2dd73d03f00" (UID: "1106b419-0b67-47af-82eb-b2dd73d03f00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:31.587202 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.587175    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-podinfo" pod "1106b419-0b67-47af-82eb-b2dd73d03f00" (UID: "1106b419-0b67-47af-82eb-b2dd73d03f00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:57:31.597562 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.597532    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-token" pod "1106b419-0b67-47af-82eb-b2dd73d03f00" (UID: "1106b419-0b67-47af-82eb-b2dd73d03f00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:57:31.722671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.713282    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:31.722671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.713328    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:31.722671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.713345    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:31.722671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.713358    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106b419-0b67-47af-82eb-b2dd73d03f00-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:31.722671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.713386    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1106b419-0b67-47af-82eb-b2dd73d03f00-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:31.722671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.713400    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1106b419-0b67-47af-82eb-b2dd73d03f00-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:31.722671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:31.713414    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27g4d\" (UniqueName: \"kubernetes.io/projected/1106b419-0b67-47af-82eb-b2dd73d03f00-kube-api-access-27g4d\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:32.196018 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:32.195978    4421 generic.go:356] "Generic (PLEG): container finished" podID="1106b419-0b67-47af-82eb-b2dd73d03f00" containerID="a1994fe1b41c29d145b1822bf32d11cd21a1ff32e82a68caa13eabd51088449a" exitCode=0
Nov 03 20:57:32.196227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:32.196036    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"
Nov 03 20:57:32.196227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:32.196077    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg" event={"ID":"1106b419-0b67-47af-82eb-b2dd73d03f00","Type":"ContainerDied","Data":"a1994fe1b41c29d145b1822bf32d11cd21a1ff32e82a68caa13eabd51088449a"}
Nov 03 20:57:32.196227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:32.196143    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg" event={"ID":"1106b419-0b67-47af-82eb-b2dd73d03f00","Type":"ContainerDied","Data":"e0214d7c6a952a35ff07cf4cf7774e3186a4d401711d1601ab4d6e1c0d3f3136"}
Nov 03 20:57:32.196227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:32.196174    4421 scope.go:122] "RemoveContainer" containerID="a1994fe1b41c29d145b1822bf32d11cd21a1ff32e82a68caa13eabd51088449a"
Nov 03 20:57:32.208370 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:32.208351    4421 scope.go:122] "RemoveContainer" containerID="abe80340a55eb2c611ebfe12bd352e7a85f0fb37a8e713be1b8006e4f9a336ec"
Nov 03 20:57:32.220635 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:32.220586    4421 scope.go:122] "RemoveContainer" containerID="a1994fe1b41c29d145b1822bf32d11cd21a1ff32e82a68caa13eabd51088449a"
Nov 03 20:57:32.222580 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:32.222556    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"]
Nov 03 20:57:32.223125 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:57:32.223104    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1994fe1b41c29d145b1822bf32d11cd21a1ff32e82a68caa13eabd51088449a\": container with ID starting with a1994fe1b41c29d145b1822bf32d11cd21a1ff32e82a68caa13eabd51088449a not found: ID does not exist" containerID="a1994fe1b41c29d145b1822bf32d11cd21a1ff32e82a68caa13eabd51088449a"
Nov 03 20:57:32.223284 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:32.223136    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1994fe1b41c29d145b1822bf32d11cd21a1ff32e82a68caa13eabd51088449a"} err="failed to get container status \"a1994fe1b41c29d145b1822bf32d11cd21a1ff32e82a68caa13eabd51088449a\": rpc error: code = NotFound desc = could not find container \"a1994fe1b41c29d145b1822bf32d11cd21a1ff32e82a68caa13eabd51088449a\": container with ID starting with a1994fe1b41c29d145b1822bf32d11cd21a1ff32e82a68caa13eabd51088449a not found: ID does not exist"
Nov 03 20:57:32.223284 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:32.223161    4421 scope.go:122] "RemoveContainer" containerID="abe80340a55eb2c611ebfe12bd352e7a85f0fb37a8e713be1b8006e4f9a336ec"
Nov 03 20:57:32.223531 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:57:32.223511    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abe80340a55eb2c611ebfe12bd352e7a85f0fb37a8e713be1b8006e4f9a336ec\": container with ID starting with abe80340a55eb2c611ebfe12bd352e7a85f0fb37a8e713be1b8006e4f9a336ec not found: ID does not exist" containerID="abe80340a55eb2c611ebfe12bd352e7a85f0fb37a8e713be1b8006e4f9a336ec"
Nov 03 20:57:32.223675 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:32.223539    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abe80340a55eb2c611ebfe12bd352e7a85f0fb37a8e713be1b8006e4f9a336ec"} err="failed to get container status \"abe80340a55eb2c611ebfe12bd352e7a85f0fb37a8e713be1b8006e4f9a336ec\": rpc error: code = NotFound desc = could not find container \"abe80340a55eb2c611ebfe12bd352e7a85f0fb37a8e713be1b8006e4f9a336ec\": container with ID starting with abe80340a55eb2c611ebfe12bd352e7a85f0fb37a8e713be1b8006e4f9a336ec not found: ID does not exist"
Nov 03 20:57:32.227091 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:32.227070    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f3a4c29a-openshift-default-b65b8b6c7-sp7qg"]
Nov 03 20:57:32.611622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:32.607774    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"]
Nov 03 20:57:32.611622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:32.608078    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn" podUID="d31c0deb-63e5-42a2-a34c-90499e741b2d" containerName="istio-proxy" containerID="cri-o://3d7ad5b4d2a608a03587b6ced12f2d778ac20ec3be2a9be4a0f18997bceed0e4" gracePeriod=30
Nov 03 20:57:34.017980 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:34.017952    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1106b419-0b67-47af-82eb-b2dd73d03f00" path="/var/lib/kubelet/pods/1106b419-0b67-47af-82eb-b2dd73d03f00/volumes"
Nov 03 20:57:36.855538 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:36.855503    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:57:36.855948 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:36.855564    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn" podUID="d31c0deb-63e5-42a2-a34c-90499e741b2d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:57:37.834000 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.833976    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:57:37.849093 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.849063    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-credential-socket\") pod \"d31c0deb-63e5-42a2-a34c-90499e741b2d\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") "
Nov 03 20:57:37.849252 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.849164    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-data\") pod \"d31c0deb-63e5-42a2-a34c-90499e741b2d\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") "
Nov 03 20:57:37.849307 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.849269    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d31c0deb-63e5-42a2-a34c-90499e741b2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31c0deb-63e5-42a2-a34c-90499e741b2d-istiod-ca-cert\") pod \"d31c0deb-63e5-42a2-a34c-90499e741b2d\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") "
Nov 03 20:57:37.849360 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.849313    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-credential-socket" pod "d31c0deb-63e5-42a2-a34c-90499e741b2d" (UID: "d31c0deb-63e5-42a2-a34c-90499e741b2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:37.849360 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.849331    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-workload-socket\") pod \"d31c0deb-63e5-42a2-a34c-90499e741b2d\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") "
Nov 03 20:57:37.849483 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.849395    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-data" pod "d31c0deb-63e5-42a2-a34c-90499e741b2d" (UID: "d31c0deb-63e5-42a2-a34c-90499e741b2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:37.849483 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.849454    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-token\" (UniqueName: \"kubernetes.io/projected/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-token\") pod \"d31c0deb-63e5-42a2-a34c-90499e741b2d\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") "
Nov 03 20:57:37.849642 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.849494    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-podinfo\") pod \"d31c0deb-63e5-42a2-a34c-90499e741b2d\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") "
Nov 03 20:57:37.849642 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.849520    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-envoy\") pod \"d31c0deb-63e5-42a2-a34c-90499e741b2d\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") "
Nov 03 20:57:37.849642 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.849576    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-workload-certs\") pod \"d31c0deb-63e5-42a2-a34c-90499e741b2d\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") "
Nov 03 20:57:37.849642 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.849592    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-workload-socket" pod "d31c0deb-63e5-42a2-a34c-90499e741b2d" (UID: "d31c0deb-63e5-42a2-a34c-90499e741b2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:37.849897 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.849691    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d31c0deb-63e5-42a2-a34c-90499e741b2d-kube-api-access-tpdk8\" (UniqueName: \"kubernetes.io/projected/d31c0deb-63e5-42a2-a34c-90499e741b2d-kube-api-access-tpdk8\") pod \"d31c0deb-63e5-42a2-a34c-90499e741b2d\" (UID: \"d31c0deb-63e5-42a2-a34c-90499e741b2d\") "
Nov 03 20:57:37.849897 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.849714    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31c0deb-63e5-42a2-a34c-90499e741b2d-istiod-ca-cert" pod "d31c0deb-63e5-42a2-a34c-90499e741b2d" (UID: "d31c0deb-63e5-42a2-a34c-90499e741b2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:57:37.850031 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.850010    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-workload-certs" pod "d31c0deb-63e5-42a2-a34c-90499e741b2d" (UID: "d31c0deb-63e5-42a2-a34c-90499e741b2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:37.850456 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.850363    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:37.850456 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.850390    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:37.850456 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.850405    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:37.850456 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.850420    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31c0deb-63e5-42a2-a34c-90499e741b2d-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:37.850456 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.850437    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:37.852482 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.852456    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-podinfo" pod "d31c0deb-63e5-42a2-a34c-90499e741b2d" (UID: "d31c0deb-63e5-42a2-a34c-90499e741b2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:57:37.852588 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.852491    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-envoy" pod "d31c0deb-63e5-42a2-a34c-90499e741b2d" (UID: "d31c0deb-63e5-42a2-a34c-90499e741b2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:57:37.852686 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.852659    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-token" pod "d31c0deb-63e5-42a2-a34c-90499e741b2d" (UID: "d31c0deb-63e5-42a2-a34c-90499e741b2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:57:37.852758 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.852733    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31c0deb-63e5-42a2-a34c-90499e741b2d-kube-api-access-tpdk8" pod "d31c0deb-63e5-42a2-a34c-90499e741b2d" (UID: "d31c0deb-63e5-42a2-a34c-90499e741b2d"). InnerVolumeSpecName "kube-api-access-tpdk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:57:37.951657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.951622    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpdk8\" (UniqueName: \"kubernetes.io/projected/d31c0deb-63e5-42a2-a34c-90499e741b2d-kube-api-access-tpdk8\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:37.951657 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.951657    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:37.952076 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.951668    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:37.952076 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:37.951677    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31c0deb-63e5-42a2-a34c-90499e741b2d-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:38.270554 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:38.270459    4421 generic.go:356] "Generic (PLEG): container finished" podID="d31c0deb-63e5-42a2-a34c-90499e741b2d" containerID="3d7ad5b4d2a608a03587b6ced12f2d778ac20ec3be2a9be4a0f18997bceed0e4" exitCode=0
Nov 03 20:57:38.270554 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:38.270504    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"
Nov 03 20:57:38.270554 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:38.270545    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn" event={"ID":"d31c0deb-63e5-42a2-a34c-90499e741b2d","Type":"ContainerDied","Data":"3d7ad5b4d2a608a03587b6ced12f2d778ac20ec3be2a9be4a0f18997bceed0e4"}
Nov 03 20:57:38.270836 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:38.270614    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn" event={"ID":"d31c0deb-63e5-42a2-a34c-90499e741b2d","Type":"ContainerDied","Data":"c22f644b6da6a0d602d01ff9ca4e9cf3bb4f90d8a3fac6aa21b9d9d0389e9326"}
Nov 03 20:57:38.270836 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:38.270637    4421 scope.go:122] "RemoveContainer" containerID="3d7ad5b4d2a608a03587b6ced12f2d778ac20ec3be2a9be4a0f18997bceed0e4"
Nov 03 20:57:38.293865 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:38.293836    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"]
Nov 03 20:57:38.299272 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:38.299252    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-47xhn"]
Nov 03 20:57:38.314380 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:38.314357    4421 scope.go:122] "RemoveContainer" containerID="3d7ad5b4d2a608a03587b6ced12f2d778ac20ec3be2a9be4a0f18997bceed0e4"
Nov 03 20:57:38.314747 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:57:38.314726    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d7ad5b4d2a608a03587b6ced12f2d778ac20ec3be2a9be4a0f18997bceed0e4\": container with ID starting with 3d7ad5b4d2a608a03587b6ced12f2d778ac20ec3be2a9be4a0f18997bceed0e4 not found: ID does not exist" containerID="3d7ad5b4d2a608a03587b6ced12f2d778ac20ec3be2a9be4a0f18997bceed0e4"
Nov 03 20:57:38.314837 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:38.314757    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d7ad5b4d2a608a03587b6ced12f2d778ac20ec3be2a9be4a0f18997bceed0e4"} err="failed to get container status \"3d7ad5b4d2a608a03587b6ced12f2d778ac20ec3be2a9be4a0f18997bceed0e4\": rpc error: code = NotFound desc = could not find container \"3d7ad5b4d2a608a03587b6ced12f2d778ac20ec3be2a9be4a0f18997bceed0e4\": container with ID starting with 3d7ad5b4d2a608a03587b6ced12f2d778ac20ec3be2a9be4a0f18997bceed0e4 not found: ID does not exist"
Nov 03 20:57:40.018612 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:57:40.018564    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d31c0deb-63e5-42a2-a34c-90499e741b2d" path="/var/lib/kubelet/pods/d31c0deb-63e5-42a2-a34c-90499e741b2d/volumes"
Nov 03 20:57:49.632914 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:57:49.632871    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:58:01.199855 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:01.199820    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j"]
Nov 03 20:58:06.201001 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:06.200951    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j" podUID="3cdc9b55-921d-4ece-baae-1134aa71f5f7" containerName="payload-processing" containerID="cri-o://d75b70b409019d5a056bd93713b3a08846b148442340e69a711db4fbb06442ba" gracePeriod=25
Nov 03 20:58:09.763847 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:09.763805    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j" podUID="3cdc9b55-921d-4ece-baae-1134aa71f5f7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:58:16.774937 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:16.774891    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj"]
Nov 03 20:58:16.784374 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:16.784341    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm"]
Nov 03 20:58:19.631923 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:58:19.631882    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:58:19.764185 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:19.764146    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j" podUID="3cdc9b55-921d-4ece-baae-1134aa71f5f7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:58:19.908413 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:19.908324    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7"]
Nov 03 20:58:19.908637 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:19.908590    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7" podUID="024fea96-e087-4d81-b1f1-1801c4ecf0ab" containerName="maas-api" containerID="cri-o://173af2a3050ac782421002cfbc321671a24e91be7aabd4cdd4d281c264673e14" gracePeriod=30
Nov 03 20:58:20.390328 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:20.390304    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7"
Nov 03 20:58:20.466911 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:20.466874    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024fea96-e087-4d81-b1f1-1801c4ecf0ab-kube-api-access-6m488\" (UniqueName: \"kubernetes.io/projected/024fea96-e087-4d81-b1f1-1801c4ecf0ab-kube-api-access-6m488\") pod \"024fea96-e087-4d81-b1f1-1801c4ecf0ab\" (UID: \"024fea96-e087-4d81-b1f1-1801c4ecf0ab\") "
Nov 03 20:58:20.467119 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:20.467006    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/024fea96-e087-4d81-b1f1-1801c4ecf0ab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/024fea96-e087-4d81-b1f1-1801c4ecf0ab-metrics-certs\") pod \"024fea96-e087-4d81-b1f1-1801c4ecf0ab\" (UID: \"024fea96-e087-4d81-b1f1-1801c4ecf0ab\") "
Nov 03 20:58:20.467119 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:20.467055    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/024fea96-e087-4d81-b1f1-1801c4ecf0ab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/024fea96-e087-4d81-b1f1-1801c4ecf0ab-openshift-service-ca\") pod \"024fea96-e087-4d81-b1f1-1801c4ecf0ab\" (UID: \"024fea96-e087-4d81-b1f1-1801c4ecf0ab\") "
Nov 03 20:58:20.467272 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:20.467198    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/024fea96-e087-4d81-b1f1-1801c4ecf0ab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/024fea96-e087-4d81-b1f1-1801c4ecf0ab-maas-api-tls\") pod \"024fea96-e087-4d81-b1f1-1801c4ecf0ab\" (UID: \"024fea96-e087-4d81-b1f1-1801c4ecf0ab\") "
Nov 03 20:58:20.467521 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:20.467488    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024fea96-e087-4d81-b1f1-1801c4ecf0ab-openshift-service-ca" pod "024fea96-e087-4d81-b1f1-1801c4ecf0ab" (UID: "024fea96-e087-4d81-b1f1-1801c4ecf0ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:58:20.468169 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:20.468147    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/024fea96-e087-4d81-b1f1-1801c4ecf0ab-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:20.470046 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:20.470001    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/024fea96-e087-4d81-b1f1-1801c4ecf0ab-maas-api-tls" pod "024fea96-e087-4d81-b1f1-1801c4ecf0ab" (UID: "024fea96-e087-4d81-b1f1-1801c4ecf0ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:58:20.470232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:20.470093    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/024fea96-e087-4d81-b1f1-1801c4ecf0ab-metrics-certs" pod "024fea96-e087-4d81-b1f1-1801c4ecf0ab" (UID: "024fea96-e087-4d81-b1f1-1801c4ecf0ab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:58:20.470791 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:20.470758    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024fea96-e087-4d81-b1f1-1801c4ecf0ab-kube-api-access-6m488" pod "024fea96-e087-4d81-b1f1-1801c4ecf0ab" (UID: "024fea96-e087-4d81-b1f1-1801c4ecf0ab"). InnerVolumeSpecName "kube-api-access-6m488". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:20.570171 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:20.570138    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m488\" (UniqueName: \"kubernetes.io/projected/024fea96-e087-4d81-b1f1-1801c4ecf0ab-kube-api-access-6m488\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:20.570171 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:20.570170    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/024fea96-e087-4d81-b1f1-1801c4ecf0ab-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:20.570375 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:20.570181    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/024fea96-e087-4d81-b1f1-1801c4ecf0ab-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:20.835234 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:20.835204    4421 generic.go:356] "Generic (PLEG): container finished" podID="024fea96-e087-4d81-b1f1-1801c4ecf0ab" containerID="173af2a3050ac782421002cfbc321671a24e91be7aabd4cdd4d281c264673e14" exitCode=0
Nov 03 20:58:20.835720 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:20.835250    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7"
Nov 03 20:58:20.835720 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:20.835290    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7" event={"ID":"024fea96-e087-4d81-b1f1-1801c4ecf0ab","Type":"ContainerDied","Data":"173af2a3050ac782421002cfbc321671a24e91be7aabd4cdd4d281c264673e14"}
Nov 03 20:58:20.835720 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:20.835342    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7" event={"ID":"024fea96-e087-4d81-b1f1-1801c4ecf0ab","Type":"ContainerDied","Data":"9f67988c83a142fb9505c2cdf5a0e76b919fa76dbe4f7b8e59dfc6ea76962253"}
Nov 03 20:58:20.835720 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:20.835366    4421 scope.go:122] "RemoveContainer" containerID="173af2a3050ac782421002cfbc321671a24e91be7aabd4cdd4d281c264673e14"
Nov 03 20:58:20.847452 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:20.847434    4421 scope.go:122] "RemoveContainer" containerID="173af2a3050ac782421002cfbc321671a24e91be7aabd4cdd4d281c264673e14"
Nov 03 20:58:20.847755 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:58:20.847736    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"173af2a3050ac782421002cfbc321671a24e91be7aabd4cdd4d281c264673e14\": container with ID starting with 173af2a3050ac782421002cfbc321671a24e91be7aabd4cdd4d281c264673e14 not found: ID does not exist" containerID="173af2a3050ac782421002cfbc321671a24e91be7aabd4cdd4d281c264673e14"
Nov 03 20:58:20.847850 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:20.847763    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"173af2a3050ac782421002cfbc321671a24e91be7aabd4cdd4d281c264673e14"} err="failed to get container status \"173af2a3050ac782421002cfbc321671a24e91be7aabd4cdd4d281c264673e14\": rpc error: code = NotFound desc = could not find container \"173af2a3050ac782421002cfbc321671a24e91be7aabd4cdd4d281c264673e14\": container with ID starting with 173af2a3050ac782421002cfbc321671a24e91be7aabd4cdd4d281c264673e14 not found: ID does not exist"
Nov 03 20:58:20.860921 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:20.860896    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7"]
Nov 03 20:58:20.864893 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:20.864871    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8d0fa-787cc57cb9-t45z7"]
Nov 03 20:58:21.775849 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:21.775799    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj" podUID="d4cc153a-39be-4e4f-bd4e-0fa01599cc56" containerName="payload-processing" containerID="cri-o://bde111a976467874c07408ca2255b1db946ceb6de3f8f82592f39a798131333b" gracePeriod=25
Nov 03 20:58:21.785156 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:21.785123    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm" podUID="2fcd696c-4d3c-484f-84e7-c6b9b7f05005" containerName="payload-pre-processing" containerID="cri-o://e237a79e683ee5d47236d079930759e4e611b12649cdb8b31968879e88c59172" gracePeriod=25
Nov 03 20:58:22.021504 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:22.021471    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="024fea96-e087-4d81-b1f1-1801c4ecf0ab" path="/var/lib/kubelet/pods/024fea96-e087-4d81-b1f1-1801c4ecf0ab/volumes"
Nov 03 20:58:22.084671 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:22.084645    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj"
Nov 03 20:58:22.198976 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:22.198937    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4cc153a-39be-4e4f-bd4e-0fa01599cc56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cc153a-39be-4e4f-bd4e-0fa01599cc56-plugins-config-volume\") pod \"d4cc153a-39be-4e4f-bd4e-0fa01599cc56\" (UID: \"d4cc153a-39be-4e4f-bd4e-0fa01599cc56\") "
Nov 03 20:58:22.199182 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:22.199043    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4cc153a-39be-4e4f-bd4e-0fa01599cc56-kube-api-access-wsplv\" (UniqueName: \"kubernetes.io/projected/d4cc153a-39be-4e4f-bd4e-0fa01599cc56-kube-api-access-wsplv\") pod \"d4cc153a-39be-4e4f-bd4e-0fa01599cc56\" (UID: \"d4cc153a-39be-4e4f-bd4e-0fa01599cc56\") "
Nov 03 20:58:22.199398 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:22.199369    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4cc153a-39be-4e4f-bd4e-0fa01599cc56-plugins-config-volume" pod "d4cc153a-39be-4e4f-bd4e-0fa01599cc56" (UID: "d4cc153a-39be-4e4f-bd4e-0fa01599cc56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:58:22.199743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:22.199721    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cc153a-39be-4e4f-bd4e-0fa01599cc56-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:22.201837 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:22.201806    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4cc153a-39be-4e4f-bd4e-0fa01599cc56-kube-api-access-wsplv" pod "d4cc153a-39be-4e4f-bd4e-0fa01599cc56" (UID: "d4cc153a-39be-4e4f-bd4e-0fa01599cc56"). InnerVolumeSpecName "kube-api-access-wsplv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:22.301188 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:22.301109    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsplv\" (UniqueName: \"kubernetes.io/projected/d4cc153a-39be-4e4f-bd4e-0fa01599cc56-kube-api-access-wsplv\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:22.852193 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:22.852155    4421 generic.go:356] "Generic (PLEG): container finished" podID="d4cc153a-39be-4e4f-bd4e-0fa01599cc56" containerID="bde111a976467874c07408ca2255b1db946ceb6de3f8f82592f39a798131333b" exitCode=0
Nov 03 20:58:22.852403 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:22.852204    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj"
Nov 03 20:58:22.852403 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:22.852242    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj" event={"ID":"d4cc153a-39be-4e4f-bd4e-0fa01599cc56","Type":"ContainerDied","Data":"bde111a976467874c07408ca2255b1db946ceb6de3f8f82592f39a798131333b"}
Nov 03 20:58:22.852403 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:22.852299    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj" event={"ID":"d4cc153a-39be-4e4f-bd4e-0fa01599cc56","Type":"ContainerDied","Data":"c6f313522ddff4a5724b3215f2c15b7713fd270f0b60e41c34f72db62bc49663"}
Nov 03 20:58:22.852403 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:22.852328    4421 scope.go:122] "RemoveContainer" containerID="bde111a976467874c07408ca2255b1db946ceb6de3f8f82592f39a798131333b"
Nov 03 20:58:22.863617 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:22.863585    4421 scope.go:122] "RemoveContainer" containerID="bde111a976467874c07408ca2255b1db946ceb6de3f8f82592f39a798131333b"
Nov 03 20:58:22.863877 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:58:22.863857    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde111a976467874c07408ca2255b1db946ceb6de3f8f82592f39a798131333b\": container with ID starting with bde111a976467874c07408ca2255b1db946ceb6de3f8f82592f39a798131333b not found: ID does not exist" containerID="bde111a976467874c07408ca2255b1db946ceb6de3f8f82592f39a798131333b"
Nov 03 20:58:22.863938 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:22.863890    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde111a976467874c07408ca2255b1db946ceb6de3f8f82592f39a798131333b"} err="failed to get container status \"bde111a976467874c07408ca2255b1db946ceb6de3f8f82592f39a798131333b\": rpc error: code = NotFound desc = could not find container \"bde111a976467874c07408ca2255b1db946ceb6de3f8f82592f39a798131333b\": container with ID starting with bde111a976467874c07408ca2255b1db946ceb6de3f8f82592f39a798131333b not found: ID does not exist"
Nov 03 20:58:22.892250 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:22.892226    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj"]
Nov 03 20:58:22.898281 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:22.898252    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-745c52-5fc5bbf548-gcdzj"]
Nov 03 20:58:24.018000 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:24.017971    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4cc153a-39be-4e4f-bd4e-0fa01599cc56" path="/var/lib/kubelet/pods/d4cc153a-39be-4e4f-bd4e-0fa01599cc56/volumes"
Nov 03 20:58:26.485513 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:26.485470    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm" podUID="2fcd696c-4d3c-484f-84e7-c6b9b7f05005" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:58:26.503026 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:26.503000    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j"
Nov 03 20:58:26.583903 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:26.583865    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cdc9b55-921d-4ece-baae-1134aa71f5f7-kube-api-access-t88bt\" (UniqueName: \"kubernetes.io/projected/3cdc9b55-921d-4ece-baae-1134aa71f5f7-kube-api-access-t88bt\") pod \"3cdc9b55-921d-4ece-baae-1134aa71f5f7\" (UID: \"3cdc9b55-921d-4ece-baae-1134aa71f5f7\") "
Nov 03 20:58:26.584175 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:26.584048    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cdc9b55-921d-4ece-baae-1134aa71f5f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cdc9b55-921d-4ece-baae-1134aa71f5f7-plugins-config-volume\") pod \"3cdc9b55-921d-4ece-baae-1134aa71f5f7\" (UID: \"3cdc9b55-921d-4ece-baae-1134aa71f5f7\") "
Nov 03 20:58:26.584765 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:26.584731    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cdc9b55-921d-4ece-baae-1134aa71f5f7-plugins-config-volume" pod "3cdc9b55-921d-4ece-baae-1134aa71f5f7" (UID: "3cdc9b55-921d-4ece-baae-1134aa71f5f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:58:26.585060 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:26.585041    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cdc9b55-921d-4ece-baae-1134aa71f5f7-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:26.587407 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:26.587382    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cdc9b55-921d-4ece-baae-1134aa71f5f7-kube-api-access-t88bt" pod "3cdc9b55-921d-4ece-baae-1134aa71f5f7" (UID: "3cdc9b55-921d-4ece-baae-1134aa71f5f7"). InnerVolumeSpecName "kube-api-access-t88bt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:26.686910 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:26.686881    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t88bt\" (UniqueName: \"kubernetes.io/projected/3cdc9b55-921d-4ece-baae-1134aa71f5f7-kube-api-access-t88bt\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:26.944621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:26.942060    4421 generic.go:356] "Generic (PLEG): container finished" podID="3cdc9b55-921d-4ece-baae-1134aa71f5f7" containerID="d75b70b409019d5a056bd93713b3a08846b148442340e69a711db4fbb06442ba" exitCode=0
Nov 03 20:58:26.944621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:26.942184    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j" event={"ID":"3cdc9b55-921d-4ece-baae-1134aa71f5f7","Type":"ContainerDied","Data":"d75b70b409019d5a056bd93713b3a08846b148442340e69a711db4fbb06442ba"}
Nov 03 20:58:26.944621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:26.942234    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j" event={"ID":"3cdc9b55-921d-4ece-baae-1134aa71f5f7","Type":"ContainerDied","Data":"1fb2beaba32181efbe45fe5833062b9c894ee7c908e3153c4741aa27ae831d5c"}
Nov 03 20:58:26.944621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:26.942263    4421 scope.go:122] "RemoveContainer" containerID="d75b70b409019d5a056bd93713b3a08846b148442340e69a711db4fbb06442ba"
Nov 03 20:58:26.944621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:26.942379    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j"
Nov 03 20:58:26.969650 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:26.969617    4421 scope.go:122] "RemoveContainer" containerID="d75b70b409019d5a056bd93713b3a08846b148442340e69a711db4fbb06442ba"
Nov 03 20:58:26.970130 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:58:26.970091    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d75b70b409019d5a056bd93713b3a08846b148442340e69a711db4fbb06442ba\": container with ID starting with d75b70b409019d5a056bd93713b3a08846b148442340e69a711db4fbb06442ba not found: ID does not exist" containerID="d75b70b409019d5a056bd93713b3a08846b148442340e69a711db4fbb06442ba"
Nov 03 20:58:26.970234 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:26.970144    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d75b70b409019d5a056bd93713b3a08846b148442340e69a711db4fbb06442ba"} err="failed to get container status \"d75b70b409019d5a056bd93713b3a08846b148442340e69a711db4fbb06442ba\": rpc error: code = NotFound desc = could not find container \"d75b70b409019d5a056bd93713b3a08846b148442340e69a711db4fbb06442ba\": container with ID starting with d75b70b409019d5a056bd93713b3a08846b148442340e69a711db4fbb06442ba not found: ID does not exist"
Nov 03 20:58:27.007917 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:27.007886    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j"]
Nov 03 20:58:27.012577 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:27.012545    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d8d0fa-5ffccdcf8-d6x9j"]
Nov 03 20:58:27.312236 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:27.312153    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv"]
Nov 03 20:58:27.312474 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:27.312433    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv" podUID="a8099016-08bd-44d8-9b1a-d586228375f0" containerName="maas-api" containerID="cri-o://6a6fafd4850bae25d390a8ae52a9a7a442182eaf57e409f65b16f92c8e79b831" gracePeriod=30
Nov 03 20:58:27.749953 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:27.749927    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv"
Nov 03 20:58:27.818451 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:27.818419    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8099016-08bd-44d8-9b1a-d586228375f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8099016-08bd-44d8-9b1a-d586228375f0-maas-api-tls\") pod \"a8099016-08bd-44d8-9b1a-d586228375f0\" (UID: \"a8099016-08bd-44d8-9b1a-d586228375f0\") "
Nov 03 20:58:27.818703 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:27.818684    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8099016-08bd-44d8-9b1a-d586228375f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8099016-08bd-44d8-9b1a-d586228375f0-openshift-service-ca\") pod \"a8099016-08bd-44d8-9b1a-d586228375f0\" (UID: \"a8099016-08bd-44d8-9b1a-d586228375f0\") "
Nov 03 20:58:27.818826 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:27.818802    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8099016-08bd-44d8-9b1a-d586228375f0-kube-api-access-4tn2g\" (UniqueName: \"kubernetes.io/projected/a8099016-08bd-44d8-9b1a-d586228375f0-kube-api-access-4tn2g\") pod \"a8099016-08bd-44d8-9b1a-d586228375f0\" (UID: \"a8099016-08bd-44d8-9b1a-d586228375f0\") "
Nov 03 20:58:27.818933 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:27.818913    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8099016-08bd-44d8-9b1a-d586228375f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8099016-08bd-44d8-9b1a-d586228375f0-metrics-certs\") pod \"a8099016-08bd-44d8-9b1a-d586228375f0\" (UID: \"a8099016-08bd-44d8-9b1a-d586228375f0\") "
Nov 03 20:58:27.819047 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:27.819018    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8099016-08bd-44d8-9b1a-d586228375f0-openshift-service-ca" pod "a8099016-08bd-44d8-9b1a-d586228375f0" (UID: "a8099016-08bd-44d8-9b1a-d586228375f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:58:27.819688 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:27.819568    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8099016-08bd-44d8-9b1a-d586228375f0-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:27.821169 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:27.821142    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8099016-08bd-44d8-9b1a-d586228375f0-maas-api-tls" pod "a8099016-08bd-44d8-9b1a-d586228375f0" (UID: "a8099016-08bd-44d8-9b1a-d586228375f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:58:27.821982 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:27.821959    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8099016-08bd-44d8-9b1a-d586228375f0-metrics-certs" pod "a8099016-08bd-44d8-9b1a-d586228375f0" (UID: "a8099016-08bd-44d8-9b1a-d586228375f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:58:27.822209 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:27.822191    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8099016-08bd-44d8-9b1a-d586228375f0-kube-api-access-4tn2g" pod "a8099016-08bd-44d8-9b1a-d586228375f0" (UID: "a8099016-08bd-44d8-9b1a-d586228375f0"). InnerVolumeSpecName "kube-api-access-4tn2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:27.921842 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:27.921800    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tn2g\" (UniqueName: \"kubernetes.io/projected/a8099016-08bd-44d8-9b1a-d586228375f0-kube-api-access-4tn2g\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:27.921842 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:27.921841    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8099016-08bd-44d8-9b1a-d586228375f0-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:27.921842 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:27.921853    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8099016-08bd-44d8-9b1a-d586228375f0-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:27.950422 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:27.950389    4421 generic.go:356] "Generic (PLEG): container finished" podID="a8099016-08bd-44d8-9b1a-d586228375f0" containerID="6a6fafd4850bae25d390a8ae52a9a7a442182eaf57e409f65b16f92c8e79b831" exitCode=0
Nov 03 20:58:27.950656 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:27.950437    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv"
Nov 03 20:58:27.950656 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:27.950459    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv" event={"ID":"a8099016-08bd-44d8-9b1a-d586228375f0","Type":"ContainerDied","Data":"6a6fafd4850bae25d390a8ae52a9a7a442182eaf57e409f65b16f92c8e79b831"}
Nov 03 20:58:27.950656 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:27.950510    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv" event={"ID":"a8099016-08bd-44d8-9b1a-d586228375f0","Type":"ContainerDied","Data":"68f488badf2206aadfd5b1ceddf5ee74cb890cff230488a845a06e09eeab93a8"}
Nov 03 20:58:27.950656 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:27.950531    4421 scope.go:122] "RemoveContainer" containerID="6a6fafd4850bae25d390a8ae52a9a7a442182eaf57e409f65b16f92c8e79b831"
Nov 03 20:58:27.962036 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:27.962020    4421 scope.go:122] "RemoveContainer" containerID="6a6fafd4850bae25d390a8ae52a9a7a442182eaf57e409f65b16f92c8e79b831"
Nov 03 20:58:27.962290 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:58:27.962274    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a6fafd4850bae25d390a8ae52a9a7a442182eaf57e409f65b16f92c8e79b831\": container with ID starting with 6a6fafd4850bae25d390a8ae52a9a7a442182eaf57e409f65b16f92c8e79b831 not found: ID does not exist" containerID="6a6fafd4850bae25d390a8ae52a9a7a442182eaf57e409f65b16f92c8e79b831"
Nov 03 20:58:27.962331 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:27.962298    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a6fafd4850bae25d390a8ae52a9a7a442182eaf57e409f65b16f92c8e79b831"} err="failed to get container status \"6a6fafd4850bae25d390a8ae52a9a7a442182eaf57e409f65b16f92c8e79b831\": rpc error: code = NotFound desc = could not find container \"6a6fafd4850bae25d390a8ae52a9a7a442182eaf57e409f65b16f92c8e79b831\": container with ID starting with 6a6fafd4850bae25d390a8ae52a9a7a442182eaf57e409f65b16f92c8e79b831 not found: ID does not exist"
Nov 03 20:58:27.978118 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:27.978092    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv"]
Nov 03 20:58:27.984222 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:27.984204    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-745c52-9cc597848-p5mqv"]
Nov 03 20:58:28.018700 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:28.018673    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cdc9b55-921d-4ece-baae-1134aa71f5f7" path="/var/lib/kubelet/pods/3cdc9b55-921d-4ece-baae-1134aa71f5f7/volumes"
Nov 03 20:58:28.019160 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:28.019146    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8099016-08bd-44d8-9b1a-d586228375f0" path="/var/lib/kubelet/pods/a8099016-08bd-44d8-9b1a-d586228375f0/volumes"
Nov 03 20:58:36.485319 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:36.485283    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm" podUID="2fcd696c-4d3c-484f-84e7-c6b9b7f05005" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:58:39.948660 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:39.948620    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k"]
Nov 03 20:58:39.961898 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:39.961866    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78"]
Nov 03 20:58:41.704506 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.704471    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l"]
Nov 03 20:58:41.705377 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705358    4421 cpu_manager.go:414] "containerMap: removing container" podUID="f399b4d8-6505-4da5-a9d2-e772cc7071d2" containerName="payload-pre-processing"
Nov 03 20:58:41.705463 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705378    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f399b4d8-6505-4da5-a9d2-e772cc7071d2" containerName="payload-pre-processing"
Nov 03 20:58:41.705463 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705397    4421 cpu_manager.go:414] "containerMap: removing container" podUID="a8099016-08bd-44d8-9b1a-d586228375f0" containerName="maas-api"
Nov 03 20:58:41.705463 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705403    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8099016-08bd-44d8-9b1a-d586228375f0" containerName="maas-api"
Nov 03 20:58:41.705463 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705418    4421 cpu_manager.go:414] "containerMap: removing container" podUID="1106b419-0b67-47af-82eb-b2dd73d03f00" containerName="istio-proxy"
Nov 03 20:58:41.705463 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705423    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1106b419-0b67-47af-82eb-b2dd73d03f00" containerName="istio-proxy"
Nov 03 20:58:41.705463 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705432    4421 cpu_manager.go:414] "containerMap: removing container" podUID="d31c0deb-63e5-42a2-a34c-90499e741b2d" containerName="istio-proxy"
Nov 03 20:58:41.705463 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705437    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d31c0deb-63e5-42a2-a34c-90499e741b2d" containerName="istio-proxy"
Nov 03 20:58:41.705463 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705443    4421 cpu_manager.go:414] "containerMap: removing container" podUID="d4cc153a-39be-4e4f-bd4e-0fa01599cc56" containerName="payload-processing"
Nov 03 20:58:41.705463 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705448    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4cc153a-39be-4e4f-bd4e-0fa01599cc56" containerName="payload-processing"
Nov 03 20:58:41.706232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705471    4421 cpu_manager.go:414] "containerMap: removing container" podUID="d5b6a788-2a4f-4819-8943-5fe88488888c" containerName="istio-proxy"
Nov 03 20:58:41.706232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705476    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5b6a788-2a4f-4819-8943-5fe88488888c" containerName="istio-proxy"
Nov 03 20:58:41.706232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705484    4421 cpu_manager.go:414] "containerMap: removing container" podUID="ca13e385-d5ab-41fe-87ce-7d903a01ee43" containerName="payload-pre-processing"
Nov 03 20:58:41.706232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705489    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca13e385-d5ab-41fe-87ce-7d903a01ee43" containerName="payload-pre-processing"
Nov 03 20:58:41.706232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705496    4421 cpu_manager.go:414] "containerMap: removing container" podUID="6e96d27c-9c01-4e0b-b036-626ad4c4c08b" containerName="payload-processing"
Nov 03 20:58:41.706232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705500    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e96d27c-9c01-4e0b-b036-626ad4c4c08b" containerName="payload-processing"
Nov 03 20:58:41.706232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705509    4421 cpu_manager.go:414] "containerMap: removing container" podUID="024fea96-e087-4d81-b1f1-1801c4ecf0ab" containerName="maas-api"
Nov 03 20:58:41.706232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705514    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="024fea96-e087-4d81-b1f1-1801c4ecf0ab" containerName="maas-api"
Nov 03 20:58:41.706232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705519    4421 cpu_manager.go:414] "containerMap: removing container" podUID="153d1bbf-3857-43c6-b66e-6dca99b48073" containerName="istio-proxy"
Nov 03 20:58:41.706232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705525    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="153d1bbf-3857-43c6-b66e-6dca99b48073" containerName="istio-proxy"
Nov 03 20:58:41.706232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705535    4421 cpu_manager.go:414] "containerMap: removing container" podUID="1e5d4c6c-d6ce-4a10-b875-6016aaf24e65" containerName="istio-proxy"
Nov 03 20:58:41.706232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705539    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e5d4c6c-d6ce-4a10-b875-6016aaf24e65" containerName="istio-proxy"
Nov 03 20:58:41.706232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705551    4421 cpu_manager.go:414] "containerMap: removing container" podUID="3cdc9b55-921d-4ece-baae-1134aa71f5f7" containerName="payload-processing"
Nov 03 20:58:41.706232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705556    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cdc9b55-921d-4ece-baae-1134aa71f5f7" containerName="payload-processing"
Nov 03 20:58:41.706232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705694    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5b6a788-2a4f-4819-8943-5fe88488888c" containerName="istio-proxy"
Nov 03 20:58:41.706232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705705    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca13e385-d5ab-41fe-87ce-7d903a01ee43" containerName="payload-pre-processing"
Nov 03 20:58:41.706232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705713    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e96d27c-9c01-4e0b-b036-626ad4c4c08b" containerName="payload-processing"
Nov 03 20:58:41.706232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705719    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="024fea96-e087-4d81-b1f1-1801c4ecf0ab" containerName="maas-api"
Nov 03 20:58:41.706232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705725    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="153d1bbf-3857-43c6-b66e-6dca99b48073" containerName="istio-proxy"
Nov 03 20:58:41.706232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705731    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="1106b419-0b67-47af-82eb-b2dd73d03f00" containerName="istio-proxy"
Nov 03 20:58:41.706232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705738    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e5d4c6c-d6ce-4a10-b875-6016aaf24e65" containerName="istio-proxy"
Nov 03 20:58:41.706232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705745    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cdc9b55-921d-4ece-baae-1134aa71f5f7" containerName="payload-processing"
Nov 03 20:58:41.706232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705751    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f399b4d8-6505-4da5-a9d2-e772cc7071d2" containerName="payload-pre-processing"
Nov 03 20:58:41.706232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705758    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8099016-08bd-44d8-9b1a-d586228375f0" containerName="maas-api"
Nov 03 20:58:41.706232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705765    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="1106b419-0b67-47af-82eb-b2dd73d03f00" containerName="istio-proxy"
Nov 03 20:58:41.706232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705772    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d31c0deb-63e5-42a2-a34c-90499e741b2d" containerName="istio-proxy"
Nov 03 20:58:41.706232 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.705776    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4cc153a-39be-4e4f-bd4e-0fa01599cc56" containerName="payload-processing"
Nov 03 20:58:41.709347 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.709332    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l"
Nov 03 20:58:41.715591 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.715571    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-3bcc2b-serving-cert\""
Nov 03 20:58:41.719475 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.719451    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l"]
Nov 03 20:58:41.876650 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.874388    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4b81cef-71a5-4458-97c9-f617fc049a8b-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l\" (UID: \"a4b81cef-71a5-4458-97c9-f617fc049a8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l"
Nov 03 20:58:41.876650 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.874488    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x8cx\" (UniqueName: \"kubernetes.io/projected/a4b81cef-71a5-4458-97c9-f617fc049a8b-kube-api-access-2x8cx\") pod \"maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l\" (UID: \"a4b81cef-71a5-4458-97c9-f617fc049a8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l"
Nov 03 20:58:41.876650 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.874651    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b81cef-71a5-4458-97c9-f617fc049a8b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l\" (UID: \"a4b81cef-71a5-4458-97c9-f617fc049a8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l"
Nov 03 20:58:41.876650 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.874714    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b81cef-71a5-4458-97c9-f617fc049a8b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l\" (UID: \"a4b81cef-71a5-4458-97c9-f617fc049a8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l"
Nov 03 20:58:41.977500 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.976865    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b81cef-71a5-4458-97c9-f617fc049a8b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l\" (UID: \"a4b81cef-71a5-4458-97c9-f617fc049a8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l"
Nov 03 20:58:41.977500 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.976953    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b81cef-71a5-4458-97c9-f617fc049a8b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l\" (UID: \"a4b81cef-71a5-4458-97c9-f617fc049a8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l"
Nov 03 20:58:41.977500 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.977124    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4b81cef-71a5-4458-97c9-f617fc049a8b-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l\" (UID: \"a4b81cef-71a5-4458-97c9-f617fc049a8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l"
Nov 03 20:58:41.977500 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.977181    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x8cx\" (UniqueName: \"kubernetes.io/projected/a4b81cef-71a5-4458-97c9-f617fc049a8b-kube-api-access-2x8cx\") pod \"maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l\" (UID: \"a4b81cef-71a5-4458-97c9-f617fc049a8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l"
Nov 03 20:58:41.977901 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.977757    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b81cef-71a5-4458-97c9-f617fc049a8b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l\" (UID: \"a4b81cef-71a5-4458-97c9-f617fc049a8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l"
Nov 03 20:58:41.993175 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.993112    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4b81cef-71a5-4458-97c9-f617fc049a8b-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l\" (UID: \"a4b81cef-71a5-4458-97c9-f617fc049a8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l"
Nov 03 20:58:41.998095 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:41.998068    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b81cef-71a5-4458-97c9-f617fc049a8b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l\" (UID: \"a4b81cef-71a5-4458-97c9-f617fc049a8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l"
Nov 03 20:58:42.001018 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:42.000990    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x8cx\" (UniqueName: \"kubernetes.io/projected/a4b81cef-71a5-4458-97c9-f617fc049a8b-kube-api-access-2x8cx\") pod \"maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l\" (UID: \"a4b81cef-71a5-4458-97c9-f617fc049a8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l"
Nov 03 20:58:42.025977 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:42.025943    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l"
Nov 03 20:58:42.085253 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:42.085223    4421 generic.go:356] "Generic (PLEG): container finished" podID="2fcd696c-4d3c-484f-84e7-c6b9b7f05005" containerID="e237a79e683ee5d47236d079930759e4e611b12649cdb8b31968879e88c59172" exitCode=0
Nov 03 20:58:42.085475 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:42.085458    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm" event={"ID":"2fcd696c-4d3c-484f-84e7-c6b9b7f05005","Type":"ContainerDied","Data":"e237a79e683ee5d47236d079930759e4e611b12649cdb8b31968879e88c59172"}
Nov 03 20:58:42.085557 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:42.085512    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm" event={"ID":"2fcd696c-4d3c-484f-84e7-c6b9b7f05005","Type":"ContainerDied","Data":"1928892efe71a960f5a5ce9f3ba354b940732df776f0f3857b356310f0b45c3a"}
Nov 03 20:58:42.085557 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:42.085537    4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1928892efe71a960f5a5ce9f3ba354b940732df776f0f3857b356310f0b45c3a"
Nov 03 20:58:42.096877 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:42.096852    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm"
Nov 03 20:58:42.282821 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:42.282733    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fcd696c-4d3c-484f-84e7-c6b9b7f05005-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fcd696c-4d3c-484f-84e7-c6b9b7f05005-plugins-config-volume\") pod \"2fcd696c-4d3c-484f-84e7-c6b9b7f05005\" (UID: \"2fcd696c-4d3c-484f-84e7-c6b9b7f05005\") "
Nov 03 20:58:42.283386 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:42.283054    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fcd696c-4d3c-484f-84e7-c6b9b7f05005-kube-api-access-hnz5k\" (UniqueName: \"kubernetes.io/projected/2fcd696c-4d3c-484f-84e7-c6b9b7f05005-kube-api-access-hnz5k\") pod \"2fcd696c-4d3c-484f-84e7-c6b9b7f05005\" (UID: \"2fcd696c-4d3c-484f-84e7-c6b9b7f05005\") "
Nov 03 20:58:42.283386 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:42.283193    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fcd696c-4d3c-484f-84e7-c6b9b7f05005-plugins-config-volume" pod "2fcd696c-4d3c-484f-84e7-c6b9b7f05005" (UID: "2fcd696c-4d3c-484f-84e7-c6b9b7f05005"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:58:42.283687 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:42.283659    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fcd696c-4d3c-484f-84e7-c6b9b7f05005-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:42.285478 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:42.285454    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fcd696c-4d3c-484f-84e7-c6b9b7f05005-kube-api-access-hnz5k" pod "2fcd696c-4d3c-484f-84e7-c6b9b7f05005" (UID: "2fcd696c-4d3c-484f-84e7-c6b9b7f05005"). InnerVolumeSpecName "kube-api-access-hnz5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:42.353305 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:42.353278    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l"]
Nov 03 20:58:42.354577 ip-10-0-143-48 kubenswrapper[4421]: W1103 20:58:42.354554    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b81cef_71a5_4458_97c9_f617fc049a8b.slice/crio-f317fff53e9db684b65008b75eba0f7072cb4a6ed884dd6c52a37438b88e4f21 WatchSource:0}: Error finding container f317fff53e9db684b65008b75eba0f7072cb4a6ed884dd6c52a37438b88e4f21: Status 404 returned error can't find the container with id f317fff53e9db684b65008b75eba0f7072cb4a6ed884dd6c52a37438b88e4f21
Nov 03 20:58:42.356524 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:42.356503    4421 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:58:42.384566 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:42.384541    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnz5k\" (UniqueName: \"kubernetes.io/projected/2fcd696c-4d3c-484f-84e7-c6b9b7f05005-kube-api-access-hnz5k\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:43.095662 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:43.095524    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm"
Nov 03 20:58:43.095662 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:43.095529    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l" event={"ID":"a4b81cef-71a5-4458-97c9-f617fc049a8b","Type":"ContainerStarted","Data":"a60569919b5538d4d5a42be5dfd87910d4c0ac0f9351ae673b2cb59e88ba9ea2"}
Nov 03 20:58:43.096171 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:43.095663    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l" event={"ID":"a4b81cef-71a5-4458-97c9-f617fc049a8b","Type":"ContainerStarted","Data":"f317fff53e9db684b65008b75eba0f7072cb4a6ed884dd6c52a37438b88e4f21"}
Nov 03 20:58:43.096171 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:43.095817    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l"
Nov 03 20:58:43.122690 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:43.122639    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l" podStartSLOduration=1.6460310310000001 podStartE2EDuration="2.122625911s" totalImagesPullingTime="476.59488ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:43.120094581 +0000 UTC m=+3453.253223044" watchObservedRunningTime="2026-11-03 20:58:43.122625911 +0000 UTC m=+3453.255754383"
Nov 03 20:58:43.138158 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:43.138079    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm"]
Nov 03 20:58:43.142400 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:43.142377    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-745c52-6478d48b6c-5lprm"]
Nov 03 20:58:44.046622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:44.043713    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fcd696c-4d3c-484f-84e7-c6b9b7f05005" path="/var/lib/kubelet/pods/2fcd696c-4d3c-484f-84e7-c6b9b7f05005/volumes"
Nov 03 20:58:44.949623 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:44.949537    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k" podUID="a12a350f-f3a6-4f34-b594-a29f2da298f4" containerName="payload-processing" containerID="cri-o://998ba0fcbc7201b02e43ca512c202e2fbf191a93d6e9f2d151bf6a8db8466ba8" gracePeriod=25
Nov 03 20:58:44.962868 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:44.962812    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78" podUID="53001e2c-5c08-4ad6-af62-053a80639a3f" containerName="payload-pre-processing" containerID="cri-o://b70137e914fc9269b5969e54b2dd65de1bd2316563fe5c1a22d775e8787a8861" gracePeriod=25
Nov 03 20:58:45.113858 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:45.113824    4421 generic.go:356] "Generic (PLEG): container finished" podID="a12a350f-f3a6-4f34-b594-a29f2da298f4" containerID="998ba0fcbc7201b02e43ca512c202e2fbf191a93d6e9f2d151bf6a8db8466ba8" exitCode=0
Nov 03 20:58:45.114008 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:45.113894    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k" event={"ID":"a12a350f-f3a6-4f34-b594-a29f2da298f4","Type":"ContainerDied","Data":"998ba0fcbc7201b02e43ca512c202e2fbf191a93d6e9f2d151bf6a8db8466ba8"}
Nov 03 20:58:45.259409 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:45.259386    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k"
Nov 03 20:58:45.330982 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:45.330952    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a12a350f-f3a6-4f34-b594-a29f2da298f4-kube-api-access-nv9mw\" (UniqueName: \"kubernetes.io/projected/a12a350f-f3a6-4f34-b594-a29f2da298f4-kube-api-access-nv9mw\") pod \"a12a350f-f3a6-4f34-b594-a29f2da298f4\" (UID: \"a12a350f-f3a6-4f34-b594-a29f2da298f4\") "
Nov 03 20:58:45.331187 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:45.331022    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a12a350f-f3a6-4f34-b594-a29f2da298f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a12a350f-f3a6-4f34-b594-a29f2da298f4-plugins-config-volume\") pod \"a12a350f-f3a6-4f34-b594-a29f2da298f4\" (UID: \"a12a350f-f3a6-4f34-b594-a29f2da298f4\") "
Nov 03 20:58:45.331752 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:45.331721    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a12a350f-f3a6-4f34-b594-a29f2da298f4-plugins-config-volume" pod "a12a350f-f3a6-4f34-b594-a29f2da298f4" (UID: "a12a350f-f3a6-4f34-b594-a29f2da298f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:58:45.332157 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:45.332132    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a12a350f-f3a6-4f34-b594-a29f2da298f4-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:45.333672 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:45.333641    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a12a350f-f3a6-4f34-b594-a29f2da298f4-kube-api-access-nv9mw" pod "a12a350f-f3a6-4f34-b594-a29f2da298f4" (UID: "a12a350f-f3a6-4f34-b594-a29f2da298f4"). InnerVolumeSpecName "kube-api-access-nv9mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:45.447326 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:45.441994    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv9mw\" (UniqueName: \"kubernetes.io/projected/a12a350f-f3a6-4f34-b594-a29f2da298f4-kube-api-access-nv9mw\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:46.145625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:46.145002    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k" event={"ID":"a12a350f-f3a6-4f34-b594-a29f2da298f4","Type":"ContainerDied","Data":"07ad98b9d744246147194a28790fbd1dfe7a572631f4756bdce83a0ff9fa6c52"}
Nov 03 20:58:46.145625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:46.145076    4421 scope.go:122] "RemoveContainer" containerID="998ba0fcbc7201b02e43ca512c202e2fbf191a93d6e9f2d151bf6a8db8466ba8"
Nov 03 20:58:46.145625 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:46.145260    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k"
Nov 03 20:58:46.172096 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:46.172066    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k"]
Nov 03 20:58:46.175830 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:46.175799    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5ecee8-5bcc57bdb4-ghl9k"]
Nov 03 20:58:46.585930 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:46.585898    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d"]
Nov 03 20:58:46.586421 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:46.586386    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d" podUID="9c1a8f22-edce-40d8-8216-adb79f5cd692" containerName="maas-api" containerID="cri-o://599deb32f0dc7c836d8a6749bb973c74e8e4d6fbed20a55a4f9f9d23bb11e7a9" gracePeriod=30
Nov 03 20:58:47.011088 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:47.011064    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d"
Nov 03 20:58:47.183303 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:47.183262    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1a8f22-edce-40d8-8216-adb79f5cd692-kube-api-access-2xhhc\" (UniqueName: \"kubernetes.io/projected/9c1a8f22-edce-40d8-8216-adb79f5cd692-kube-api-access-2xhhc\") pod \"9c1a8f22-edce-40d8-8216-adb79f5cd692\" (UID: \"9c1a8f22-edce-40d8-8216-adb79f5cd692\") "
Nov 03 20:58:47.183924 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:47.183316    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c1a8f22-edce-40d8-8216-adb79f5cd692-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1a8f22-edce-40d8-8216-adb79f5cd692-openshift-service-ca\") pod \"9c1a8f22-edce-40d8-8216-adb79f5cd692\" (UID: \"9c1a8f22-edce-40d8-8216-adb79f5cd692\") "
Nov 03 20:58:47.183924 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:47.183397    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c1a8f22-edce-40d8-8216-adb79f5cd692-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c1a8f22-edce-40d8-8216-adb79f5cd692-maas-api-tls\") pod \"9c1a8f22-edce-40d8-8216-adb79f5cd692\" (UID: \"9c1a8f22-edce-40d8-8216-adb79f5cd692\") "
Nov 03 20:58:47.183924 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:47.183460    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c1a8f22-edce-40d8-8216-adb79f5cd692-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c1a8f22-edce-40d8-8216-adb79f5cd692-metrics-certs\") pod \"9c1a8f22-edce-40d8-8216-adb79f5cd692\" (UID: \"9c1a8f22-edce-40d8-8216-adb79f5cd692\") "
Nov 03 20:58:47.184500 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:47.184467    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1a8f22-edce-40d8-8216-adb79f5cd692-openshift-service-ca" pod "9c1a8f22-edce-40d8-8216-adb79f5cd692" (UID: "9c1a8f22-edce-40d8-8216-adb79f5cd692"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:58:47.185410 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:47.185232    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1a8f22-edce-40d8-8216-adb79f5cd692-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:47.186357 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:47.186326    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c1a8f22-edce-40d8-8216-adb79f5cd692-maas-api-tls" pod "9c1a8f22-edce-40d8-8216-adb79f5cd692" (UID: "9c1a8f22-edce-40d8-8216-adb79f5cd692"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:58:47.186507 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:47.186489    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c1a8f22-edce-40d8-8216-adb79f5cd692-metrics-certs" pod "9c1a8f22-edce-40d8-8216-adb79f5cd692" (UID: "9c1a8f22-edce-40d8-8216-adb79f5cd692"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:58:47.187163 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:47.187142    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1a8f22-edce-40d8-8216-adb79f5cd692-kube-api-access-2xhhc" pod "9c1a8f22-edce-40d8-8216-adb79f5cd692" (UID: "9c1a8f22-edce-40d8-8216-adb79f5cd692"). InnerVolumeSpecName "kube-api-access-2xhhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:47.190990 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:47.190966    4421 generic.go:356] "Generic (PLEG): container finished" podID="9c1a8f22-edce-40d8-8216-adb79f5cd692" containerID="599deb32f0dc7c836d8a6749bb973c74e8e4d6fbed20a55a4f9f9d23bb11e7a9" exitCode=0
Nov 03 20:58:47.191120 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:47.191020    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d" event={"ID":"9c1a8f22-edce-40d8-8216-adb79f5cd692","Type":"ContainerDied","Data":"599deb32f0dc7c836d8a6749bb973c74e8e4d6fbed20a55a4f9f9d23bb11e7a9"}
Nov 03 20:58:47.191120 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:47.191023    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d"
Nov 03 20:58:47.191120 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:47.191061    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d" event={"ID":"9c1a8f22-edce-40d8-8216-adb79f5cd692","Type":"ContainerDied","Data":"b34c0ff64732f9bbbee617330cfae06e0daf7f3d3d3e7741b6dbd9421e504150"}
Nov 03 20:58:47.191120 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:47.191084    4421 scope.go:122] "RemoveContainer" containerID="599deb32f0dc7c836d8a6749bb973c74e8e4d6fbed20a55a4f9f9d23bb11e7a9"
Nov 03 20:58:47.220510 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:47.220488    4421 scope.go:122] "RemoveContainer" containerID="599deb32f0dc7c836d8a6749bb973c74e8e4d6fbed20a55a4f9f9d23bb11e7a9"
Nov 03 20:58:47.221080 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:58:47.221057    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"599deb32f0dc7c836d8a6749bb973c74e8e4d6fbed20a55a4f9f9d23bb11e7a9\": container with ID starting with 599deb32f0dc7c836d8a6749bb973c74e8e4d6fbed20a55a4f9f9d23bb11e7a9 not found: ID does not exist" containerID="599deb32f0dc7c836d8a6749bb973c74e8e4d6fbed20a55a4f9f9d23bb11e7a9"
Nov 03 20:58:47.221180 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:47.221088    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599deb32f0dc7c836d8a6749bb973c74e8e4d6fbed20a55a4f9f9d23bb11e7a9"} err="failed to get container status \"599deb32f0dc7c836d8a6749bb973c74e8e4d6fbed20a55a4f9f9d23bb11e7a9\": rpc error: code = NotFound desc = could not find container \"599deb32f0dc7c836d8a6749bb973c74e8e4d6fbed20a55a4f9f9d23bb11e7a9\": container with ID starting with 599deb32f0dc7c836d8a6749bb973c74e8e4d6fbed20a55a4f9f9d23bb11e7a9 not found: ID does not exist"
Nov 03 20:58:47.232634 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:47.232550    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d"]
Nov 03 20:58:47.236579 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:47.236556    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-03c633-7fb4fb5965-9hv4d"]
Nov 03 20:58:47.286834 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:47.286802    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xhhc\" (UniqueName: \"kubernetes.io/projected/9c1a8f22-edce-40d8-8216-adb79f5cd692-kube-api-access-2xhhc\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:47.286834 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:47.286833    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c1a8f22-edce-40d8-8216-adb79f5cd692-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:47.287000 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:47.286846    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c1a8f22-edce-40d8-8216-adb79f5cd692-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:48.018082 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:48.018040    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c1a8f22-edce-40d8-8216-adb79f5cd692" path="/var/lib/kubelet/pods/9c1a8f22-edce-40d8-8216-adb79f5cd692/volumes"
Nov 03 20:58:48.018574 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:48.018552    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a12a350f-f3a6-4f34-b594-a29f2da298f4" path="/var/lib/kubelet/pods/a12a350f-f3a6-4f34-b594-a29f2da298f4/volumes"
Nov 03 20:58:49.109091 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:49.109062    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l"
Nov 03 20:58:49.631732 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:58:49.631686    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:58:52.388013 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:52.387976    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78" podUID="53001e2c-5c08-4ad6-af62-053a80639a3f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:58:56.381050 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:56.380992    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f"]
Nov 03 20:58:56.401503 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:58:56.401465    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz"]
Nov 03 20:59:01.381806 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:01.381763    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f" podUID="e5fdfb8c-95f4-47c8-8154-4864c5485f24" containerName="payload-processing" containerID="cri-o://321e25ca960e84c979fba6dce522d466156d2dfba9d597d60e50437f0e40c0ab" gracePeriod=25
Nov 03 20:59:01.402680 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:01.402646    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz" podUID="8fdce1ab-1499-40bf-8f67-f1f23b2959b8" containerName="payload-pre-processing" containerID="cri-o://58e383371f8740126034176d160b5f6654e2d22c13387627dd5e07bdbebe4d9f" gracePeriod=25
Nov 03 20:59:01.691309 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:01.691282    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f"
Nov 03 20:59:01.765833 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:01.765797    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5fdfb8c-95f4-47c8-8154-4864c5485f24-kube-api-access-5ckjd\" (UniqueName: \"kubernetes.io/projected/e5fdfb8c-95f4-47c8-8154-4864c5485f24-kube-api-access-5ckjd\") pod \"e5fdfb8c-95f4-47c8-8154-4864c5485f24\" (UID: \"e5fdfb8c-95f4-47c8-8154-4864c5485f24\") "
Nov 03 20:59:01.766152 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:01.766124    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5fdfb8c-95f4-47c8-8154-4864c5485f24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5fdfb8c-95f4-47c8-8154-4864c5485f24-plugins-config-volume\") pod \"e5fdfb8c-95f4-47c8-8154-4864c5485f24\" (UID: \"e5fdfb8c-95f4-47c8-8154-4864c5485f24\") "
Nov 03 20:59:01.766553 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:01.766528    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5fdfb8c-95f4-47c8-8154-4864c5485f24-plugins-config-volume" pod "e5fdfb8c-95f4-47c8-8154-4864c5485f24" (UID: "e5fdfb8c-95f4-47c8-8154-4864c5485f24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:59:01.766779 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:01.766762    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5fdfb8c-95f4-47c8-8154-4864c5485f24-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:01.769090 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:01.769052    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5fdfb8c-95f4-47c8-8154-4864c5485f24-kube-api-access-5ckjd" pod "e5fdfb8c-95f4-47c8-8154-4864c5485f24" (UID: "e5fdfb8c-95f4-47c8-8154-4864c5485f24"). InnerVolumeSpecName "kube-api-access-5ckjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:01.773350 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:01.773330    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz"
Nov 03 20:59:01.867960 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:01.867930    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fdce1ab-1499-40bf-8f67-f1f23b2959b8-kube-api-access-s5cfn\" (UniqueName: \"kubernetes.io/projected/8fdce1ab-1499-40bf-8f67-f1f23b2959b8-kube-api-access-s5cfn\") pod \"8fdce1ab-1499-40bf-8f67-f1f23b2959b8\" (UID: \"8fdce1ab-1499-40bf-8f67-f1f23b2959b8\") "
Nov 03 20:59:01.868152 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:01.867970    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fdce1ab-1499-40bf-8f67-f1f23b2959b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fdce1ab-1499-40bf-8f67-f1f23b2959b8-plugins-config-volume\") pod \"8fdce1ab-1499-40bf-8f67-f1f23b2959b8\" (UID: \"8fdce1ab-1499-40bf-8f67-f1f23b2959b8\") "
Nov 03 20:59:01.868390 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:01.868372    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ckjd\" (UniqueName: \"kubernetes.io/projected/e5fdfb8c-95f4-47c8-8154-4864c5485f24-kube-api-access-5ckjd\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:01.868466 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:01.868384    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fdce1ab-1499-40bf-8f67-f1f23b2959b8-plugins-config-volume" pod "8fdce1ab-1499-40bf-8f67-f1f23b2959b8" (UID: "8fdce1ab-1499-40bf-8f67-f1f23b2959b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:59:01.870135 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:01.870114    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fdce1ab-1499-40bf-8f67-f1f23b2959b8-kube-api-access-s5cfn" pod "8fdce1ab-1499-40bf-8f67-f1f23b2959b8" (UID: "8fdce1ab-1499-40bf-8f67-f1f23b2959b8"). InnerVolumeSpecName "kube-api-access-s5cfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:01.970243 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:01.970201    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5cfn\" (UniqueName: \"kubernetes.io/projected/8fdce1ab-1499-40bf-8f67-f1f23b2959b8-kube-api-access-s5cfn\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:01.970243 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:01.970236    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fdce1ab-1499-40bf-8f67-f1f23b2959b8-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:02.419830 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:02.419789    4421 generic.go:356] "Generic (PLEG): container finished" podID="8fdce1ab-1499-40bf-8f67-f1f23b2959b8" containerID="58e383371f8740126034176d160b5f6654e2d22c13387627dd5e07bdbebe4d9f" exitCode=0
Nov 03 20:59:02.420320 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:02.419917    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz" event={"ID":"8fdce1ab-1499-40bf-8f67-f1f23b2959b8","Type":"ContainerDied","Data":"58e383371f8740126034176d160b5f6654e2d22c13387627dd5e07bdbebe4d9f"}
Nov 03 20:59:02.420320 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:02.419970    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz" event={"ID":"8fdce1ab-1499-40bf-8f67-f1f23b2959b8","Type":"ContainerDied","Data":"5a67df43959f9888bb2a92224c786a7b96ab5f09080f348c6e043bea63abe098"}
Nov 03 20:59:02.420320 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:02.420001    4421 scope.go:122] "RemoveContainer" containerID="58e383371f8740126034176d160b5f6654e2d22c13387627dd5e07bdbebe4d9f"
Nov 03 20:59:02.420320 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:02.420174    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz"
Nov 03 20:59:02.421317 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:02.421281    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78" podUID="53001e2c-5c08-4ad6-af62-053a80639a3f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:59:02.430565 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:02.430532    4421 generic.go:356] "Generic (PLEG): container finished" podID="e5fdfb8c-95f4-47c8-8154-4864c5485f24" containerID="321e25ca960e84c979fba6dce522d466156d2dfba9d597d60e50437f0e40c0ab" exitCode=0
Nov 03 20:59:02.430751 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:02.430576    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f" event={"ID":"e5fdfb8c-95f4-47c8-8154-4864c5485f24","Type":"ContainerDied","Data":"321e25ca960e84c979fba6dce522d466156d2dfba9d597d60e50437f0e40c0ab"}
Nov 03 20:59:02.430751 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:02.430636    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f" event={"ID":"e5fdfb8c-95f4-47c8-8154-4864c5485f24","Type":"ContainerDied","Data":"8459761267dafad76199c0a959bb08cb6d0e4b8c04edd1a3d7c9d63e997243a1"}
Nov 03 20:59:02.430751 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:02.430712    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f"
Nov 03 20:59:02.445992 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:02.445972    4421 scope.go:122] "RemoveContainer" containerID="58e383371f8740126034176d160b5f6654e2d22c13387627dd5e07bdbebe4d9f"
Nov 03 20:59:02.446377 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:59:02.446344    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58e383371f8740126034176d160b5f6654e2d22c13387627dd5e07bdbebe4d9f\": container with ID starting with 58e383371f8740126034176d160b5f6654e2d22c13387627dd5e07bdbebe4d9f not found: ID does not exist" containerID="58e383371f8740126034176d160b5f6654e2d22c13387627dd5e07bdbebe4d9f"
Nov 03 20:59:02.446512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:02.446388    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58e383371f8740126034176d160b5f6654e2d22c13387627dd5e07bdbebe4d9f"} err="failed to get container status \"58e383371f8740126034176d160b5f6654e2d22c13387627dd5e07bdbebe4d9f\": rpc error: code = NotFound desc = could not find container \"58e383371f8740126034176d160b5f6654e2d22c13387627dd5e07bdbebe4d9f\": container with ID starting with 58e383371f8740126034176d160b5f6654e2d22c13387627dd5e07bdbebe4d9f not found: ID does not exist"
Nov 03 20:59:02.446512 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:02.446414    4421 scope.go:122] "RemoveContainer" containerID="321e25ca960e84c979fba6dce522d466156d2dfba9d597d60e50437f0e40c0ab"
Nov 03 20:59:02.450516 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:02.450489    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz"]
Nov 03 20:59:02.455778 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:02.455750    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3c25bf-6cdf4f84c9-r8rrz"]
Nov 03 20:59:02.472619 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:02.466006    4421 scope.go:122] "RemoveContainer" containerID="321e25ca960e84c979fba6dce522d466156d2dfba9d597d60e50437f0e40c0ab"
Nov 03 20:59:02.472769 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:02.472669    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f"]
Nov 03 20:59:02.475197 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:59:02.473134    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"321e25ca960e84c979fba6dce522d466156d2dfba9d597d60e50437f0e40c0ab\": container with ID starting with 321e25ca960e84c979fba6dce522d466156d2dfba9d597d60e50437f0e40c0ab not found: ID does not exist" containerID="321e25ca960e84c979fba6dce522d466156d2dfba9d597d60e50437f0e40c0ab"
Nov 03 20:59:02.475197 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:02.473169    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321e25ca960e84c979fba6dce522d466156d2dfba9d597d60e50437f0e40c0ab"} err="failed to get container status \"321e25ca960e84c979fba6dce522d466156d2dfba9d597d60e50437f0e40c0ab\": rpc error: code = NotFound desc = could not find container \"321e25ca960e84c979fba6dce522d466156d2dfba9d597d60e50437f0e40c0ab\": container with ID starting with 321e25ca960e84c979fba6dce522d466156d2dfba9d597d60e50437f0e40c0ab not found: ID does not exist"
Nov 03 20:59:02.477967 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:02.477880    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3c25bf-664f9598-fzb5f"]
Nov 03 20:59:04.042743 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:04.042708    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fdce1ab-1499-40bf-8f67-f1f23b2959b8" path="/var/lib/kubelet/pods/8fdce1ab-1499-40bf-8f67-f1f23b2959b8/volumes"
Nov 03 20:59:04.043404 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:04.043381    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5fdfb8c-95f4-47c8-8154-4864c5485f24" path="/var/lib/kubelet/pods/e5fdfb8c-95f4-47c8-8154-4864c5485f24/volumes"
Nov 03 20:59:05.249562 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:05.249536    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78"
Nov 03 20:59:05.337888 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:05.337835    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53001e2c-5c08-4ad6-af62-053a80639a3f-kube-api-access-x87ng\" (UniqueName: \"kubernetes.io/projected/53001e2c-5c08-4ad6-af62-053a80639a3f-kube-api-access-x87ng\") pod \"53001e2c-5c08-4ad6-af62-053a80639a3f\" (UID: \"53001e2c-5c08-4ad6-af62-053a80639a3f\") "
Nov 03 20:59:05.338194 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:05.338168    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53001e2c-5c08-4ad6-af62-053a80639a3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53001e2c-5c08-4ad6-af62-053a80639a3f-plugins-config-volume\") pod \"53001e2c-5c08-4ad6-af62-053a80639a3f\" (UID: \"53001e2c-5c08-4ad6-af62-053a80639a3f\") "
Nov 03 20:59:05.338562 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:05.338537    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53001e2c-5c08-4ad6-af62-053a80639a3f-plugins-config-volume" pod "53001e2c-5c08-4ad6-af62-053a80639a3f" (UID: "53001e2c-5c08-4ad6-af62-053a80639a3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:59:05.338816 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:05.338798    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53001e2c-5c08-4ad6-af62-053a80639a3f-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:05.340500 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:05.340478    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53001e2c-5c08-4ad6-af62-053a80639a3f-kube-api-access-x87ng" pod "53001e2c-5c08-4ad6-af62-053a80639a3f" (UID: "53001e2c-5c08-4ad6-af62-053a80639a3f"). InnerVolumeSpecName "kube-api-access-x87ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:05.445622 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:05.441140    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x87ng\" (UniqueName: \"kubernetes.io/projected/53001e2c-5c08-4ad6-af62-053a80639a3f-kube-api-access-x87ng\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:05.458617 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:05.458561    4421 generic.go:356] "Generic (PLEG): container finished" podID="53001e2c-5c08-4ad6-af62-053a80639a3f" containerID="b70137e914fc9269b5969e54b2dd65de1bd2316563fe5c1a22d775e8787a8861" exitCode=0
Nov 03 20:59:05.458764 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:05.458644    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78"
Nov 03 20:59:05.458764 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:05.458656    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78" event={"ID":"53001e2c-5c08-4ad6-af62-053a80639a3f","Type":"ContainerDied","Data":"b70137e914fc9269b5969e54b2dd65de1bd2316563fe5c1a22d775e8787a8861"}
Nov 03 20:59:05.458764 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:05.458718    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78" event={"ID":"53001e2c-5c08-4ad6-af62-053a80639a3f","Type":"ContainerDied","Data":"93cd764c50a10ac5176ffe6f3e9fa7b20ba9668cded20e2d7e69f46c497b04b2"}
Nov 03 20:59:05.458764 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:05.458747    4421 scope.go:122] "RemoveContainer" containerID="b70137e914fc9269b5969e54b2dd65de1bd2316563fe5c1a22d775e8787a8861"
Nov 03 20:59:05.470292 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:05.470271    4421 scope.go:122] "RemoveContainer" containerID="b70137e914fc9269b5969e54b2dd65de1bd2316563fe5c1a22d775e8787a8861"
Nov 03 20:59:05.470553 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:59:05.470534    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b70137e914fc9269b5969e54b2dd65de1bd2316563fe5c1a22d775e8787a8861\": container with ID starting with b70137e914fc9269b5969e54b2dd65de1bd2316563fe5c1a22d775e8787a8861 not found: ID does not exist" containerID="b70137e914fc9269b5969e54b2dd65de1bd2316563fe5c1a22d775e8787a8861"
Nov 03 20:59:05.470640 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:05.470562    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b70137e914fc9269b5969e54b2dd65de1bd2316563fe5c1a22d775e8787a8861"} err="failed to get container status \"b70137e914fc9269b5969e54b2dd65de1bd2316563fe5c1a22d775e8787a8861\": rpc error: code = NotFound desc = could not find container \"b70137e914fc9269b5969e54b2dd65de1bd2316563fe5c1a22d775e8787a8861\": container with ID starting with b70137e914fc9269b5969e54b2dd65de1bd2316563fe5c1a22d775e8787a8861 not found: ID does not exist"
Nov 03 20:59:05.483481 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:05.483421    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78"]
Nov 03 20:59:05.487245 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:05.487223    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ecee8-677dd698bc-xfv78"]
Nov 03 20:59:06.017554 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:06.017522    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53001e2c-5c08-4ad6-af62-053a80639a3f" path="/var/lib/kubelet/pods/53001e2c-5c08-4ad6-af62-053a80639a3f/volumes"
Nov 03 20:59:08.809326 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:08.809292    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p"]
Nov 03 20:59:08.809896 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:08.809567    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p" podUID="0130f392-da22-43d5-87e5-6f69d6475d7d" containerName="maas-api" containerID="cri-o://fd51a771f71f1b01a6da015c7bb9123e4acc4cbc411bb681488e1d1fc1cb6449" gracePeriod=30
Nov 03 20:59:09.227048 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:09.227022    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p"
Nov 03 20:59:09.301734 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:09.301704    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0130f392-da22-43d5-87e5-6f69d6475d7d-kube-api-access-98dps\" (UniqueName: \"kubernetes.io/projected/0130f392-da22-43d5-87e5-6f69d6475d7d-kube-api-access-98dps\") pod \"0130f392-da22-43d5-87e5-6f69d6475d7d\" (UID: \"0130f392-da22-43d5-87e5-6f69d6475d7d\") "
Nov 03 20:59:09.301963 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:09.301943    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0130f392-da22-43d5-87e5-6f69d6475d7d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0130f392-da22-43d5-87e5-6f69d6475d7d-metrics-certs\") pod \"0130f392-da22-43d5-87e5-6f69d6475d7d\" (UID: \"0130f392-da22-43d5-87e5-6f69d6475d7d\") "
Nov 03 20:59:09.302078 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:09.302052    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0130f392-da22-43d5-87e5-6f69d6475d7d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0130f392-da22-43d5-87e5-6f69d6475d7d-openshift-service-ca\") pod \"0130f392-da22-43d5-87e5-6f69d6475d7d\" (UID: \"0130f392-da22-43d5-87e5-6f69d6475d7d\") "
Nov 03 20:59:09.302181 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:09.302101    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0130f392-da22-43d5-87e5-6f69d6475d7d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0130f392-da22-43d5-87e5-6f69d6475d7d-maas-api-tls\") pod \"0130f392-da22-43d5-87e5-6f69d6475d7d\" (UID: \"0130f392-da22-43d5-87e5-6f69d6475d7d\") "
Nov 03 20:59:09.302533 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:09.302505    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0130f392-da22-43d5-87e5-6f69d6475d7d-openshift-service-ca" pod "0130f392-da22-43d5-87e5-6f69d6475d7d" (UID: "0130f392-da22-43d5-87e5-6f69d6475d7d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:59:09.302831 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:09.302806    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0130f392-da22-43d5-87e5-6f69d6475d7d-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:09.304691 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:09.304658    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0130f392-da22-43d5-87e5-6f69d6475d7d-metrics-certs" pod "0130f392-da22-43d5-87e5-6f69d6475d7d" (UID: "0130f392-da22-43d5-87e5-6f69d6475d7d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:59:09.304691 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:09.304673    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0130f392-da22-43d5-87e5-6f69d6475d7d-kube-api-access-98dps" pod "0130f392-da22-43d5-87e5-6f69d6475d7d" (UID: "0130f392-da22-43d5-87e5-6f69d6475d7d"). InnerVolumeSpecName "kube-api-access-98dps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:09.304893 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:09.304862    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0130f392-da22-43d5-87e5-6f69d6475d7d-maas-api-tls" pod "0130f392-da22-43d5-87e5-6f69d6475d7d" (UID: "0130f392-da22-43d5-87e5-6f69d6475d7d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:59:09.404964 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:09.404927    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0130f392-da22-43d5-87e5-6f69d6475d7d-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:09.404964 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:09.404960    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0130f392-da22-43d5-87e5-6f69d6475d7d-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:09.404964 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:09.404972    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98dps\" (UniqueName: \"kubernetes.io/projected/0130f392-da22-43d5-87e5-6f69d6475d7d-kube-api-access-98dps\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:09.489577 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:09.489494    4421 generic.go:356] "Generic (PLEG): container finished" podID="0130f392-da22-43d5-87e5-6f69d6475d7d" containerID="fd51a771f71f1b01a6da015c7bb9123e4acc4cbc411bb681488e1d1fc1cb6449" exitCode=0
Nov 03 20:59:09.489577 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:09.489543    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p"
Nov 03 20:59:09.489776 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:09.489574    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p" event={"ID":"0130f392-da22-43d5-87e5-6f69d6475d7d","Type":"ContainerDied","Data":"fd51a771f71f1b01a6da015c7bb9123e4acc4cbc411bb681488e1d1fc1cb6449"}
Nov 03 20:59:09.489776 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:09.489669    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p" event={"ID":"0130f392-da22-43d5-87e5-6f69d6475d7d","Type":"ContainerDied","Data":"665889268f8ca06bb46bc46e80ceeb5e31d08c7a79b5cfb7f4c0ab9a6288c2e1"}
Nov 03 20:59:09.489776 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:09.489693    4421 scope.go:122] "RemoveContainer" containerID="fd51a771f71f1b01a6da015c7bb9123e4acc4cbc411bb681488e1d1fc1cb6449"
Nov 03 20:59:09.501454 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:09.501438    4421 scope.go:122] "RemoveContainer" containerID="fd51a771f71f1b01a6da015c7bb9123e4acc4cbc411bb681488e1d1fc1cb6449"
Nov 03 20:59:09.501738 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:59:09.501717    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd51a771f71f1b01a6da015c7bb9123e4acc4cbc411bb681488e1d1fc1cb6449\": container with ID starting with fd51a771f71f1b01a6da015c7bb9123e4acc4cbc411bb681488e1d1fc1cb6449 not found: ID does not exist" containerID="fd51a771f71f1b01a6da015c7bb9123e4acc4cbc411bb681488e1d1fc1cb6449"
Nov 03 20:59:09.501816 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:09.501746    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd51a771f71f1b01a6da015c7bb9123e4acc4cbc411bb681488e1d1fc1cb6449"} err="failed to get container status \"fd51a771f71f1b01a6da015c7bb9123e4acc4cbc411bb681488e1d1fc1cb6449\": rpc error: code = NotFound desc = could not find container \"fd51a771f71f1b01a6da015c7bb9123e4acc4cbc411bb681488e1d1fc1cb6449\": container with ID starting with fd51a771f71f1b01a6da015c7bb9123e4acc4cbc411bb681488e1d1fc1cb6449 not found: ID does not exist"
Nov 03 20:59:09.514326 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:09.514287    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p"]
Nov 03 20:59:09.518301 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:09.518280    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3c25bf-7b4dff86bc-pwj4p"]
Nov 03 20:59:10.018158 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:10.018127    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0130f392-da22-43d5-87e5-6f69d6475d7d" path="/var/lib/kubelet/pods/0130f392-da22-43d5-87e5-6f69d6475d7d/volumes"
Nov 03 20:59:13.207027 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:13.207000    4421 scope.go:122] "RemoveContainer" containerID="e237a79e683ee5d47236d079930759e4e611b12649cdb8b31968879e88c59172"
Nov 03 20:59:18.696318 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:18.696282    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg"]
Nov 03 20:59:19.630118 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:59:19.630078    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:59:23.697132 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:23.697076    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg" podUID="bd5b0764-48b8-4909-b75d-8f4b8575b015" containerName="payload-processing" containerID="cri-o://b97049446b49a88bea959b1ab8e2673fe52d575369b5acefa6c1738718930d5b" gracePeriod=25
Nov 03 20:59:24.023079 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:24.023052    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg"
Nov 03 20:59:24.143029 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:24.143002    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd5b0764-48b8-4909-b75d-8f4b8575b015-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd5b0764-48b8-4909-b75d-8f4b8575b015-plugins-config-volume\") pod \"bd5b0764-48b8-4909-b75d-8f4b8575b015\" (UID: \"bd5b0764-48b8-4909-b75d-8f4b8575b015\") "
Nov 03 20:59:24.143225 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:24.143163    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd5b0764-48b8-4909-b75d-8f4b8575b015-kube-api-access-kwhkw\" (UniqueName: \"kubernetes.io/projected/bd5b0764-48b8-4909-b75d-8f4b8575b015-kube-api-access-kwhkw\") pod \"bd5b0764-48b8-4909-b75d-8f4b8575b015\" (UID: \"bd5b0764-48b8-4909-b75d-8f4b8575b015\") "
Nov 03 20:59:24.143350 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:24.143329    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd5b0764-48b8-4909-b75d-8f4b8575b015-plugins-config-volume" pod "bd5b0764-48b8-4909-b75d-8f4b8575b015" (UID: "bd5b0764-48b8-4909-b75d-8f4b8575b015"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:59:24.143747 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:24.143724    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd5b0764-48b8-4909-b75d-8f4b8575b015-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:24.145461 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:24.145438    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd5b0764-48b8-4909-b75d-8f4b8575b015-kube-api-access-kwhkw" pod "bd5b0764-48b8-4909-b75d-8f4b8575b015" (UID: "bd5b0764-48b8-4909-b75d-8f4b8575b015"). InnerVolumeSpecName "kube-api-access-kwhkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:24.245990 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:24.245895    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwhkw\" (UniqueName: \"kubernetes.io/projected/bd5b0764-48b8-4909-b75d-8f4b8575b015-kube-api-access-kwhkw\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:24.600465 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:24.600375    4421 generic.go:356] "Generic (PLEG): container finished" podID="bd5b0764-48b8-4909-b75d-8f4b8575b015" containerID="b97049446b49a88bea959b1ab8e2673fe52d575369b5acefa6c1738718930d5b" exitCode=0
Nov 03 20:59:24.600465 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:24.600413    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg"
Nov 03 20:59:24.600465 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:24.600455    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg" event={"ID":"bd5b0764-48b8-4909-b75d-8f4b8575b015","Type":"ContainerDied","Data":"b97049446b49a88bea959b1ab8e2673fe52d575369b5acefa6c1738718930d5b"}
Nov 03 20:59:24.600793 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:24.600503    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg" event={"ID":"bd5b0764-48b8-4909-b75d-8f4b8575b015","Type":"ContainerDied","Data":"3f21ea9bc7fcb2927cb8823044849f8c671de0ff506be55522dbcd8f64a6fd75"}
Nov 03 20:59:24.600793 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:24.600526    4421 scope.go:122] "RemoveContainer" containerID="b97049446b49a88bea959b1ab8e2673fe52d575369b5acefa6c1738718930d5b"
Nov 03 20:59:24.611700 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:24.611683    4421 scope.go:122] "RemoveContainer" containerID="b97049446b49a88bea959b1ab8e2673fe52d575369b5acefa6c1738718930d5b"
Nov 03 20:59:24.612041 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:59:24.612020    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b97049446b49a88bea959b1ab8e2673fe52d575369b5acefa6c1738718930d5b\": container with ID starting with b97049446b49a88bea959b1ab8e2673fe52d575369b5acefa6c1738718930d5b not found: ID does not exist" containerID="b97049446b49a88bea959b1ab8e2673fe52d575369b5acefa6c1738718930d5b"
Nov 03 20:59:24.612096 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:24.612048    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b97049446b49a88bea959b1ab8e2673fe52d575369b5acefa6c1738718930d5b"} err="failed to get container status \"b97049446b49a88bea959b1ab8e2673fe52d575369b5acefa6c1738718930d5b\": rpc error: code = NotFound desc = could not find container \"b97049446b49a88bea959b1ab8e2673fe52d575369b5acefa6c1738718930d5b\": container with ID starting with b97049446b49a88bea959b1ab8e2673fe52d575369b5acefa6c1738718930d5b not found: ID does not exist"
Nov 03 20:59:24.625355 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:24.625332    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg"]
Nov 03 20:59:24.628761 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:24.628741    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-30e9de-54f79c474b-dh4jg"]
Nov 03 20:59:26.017481 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:26.017445    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd5b0764-48b8-4909-b75d-8f4b8575b015" path="/var/lib/kubelet/pods/bd5b0764-48b8-4909-b75d-8f4b8575b015/volumes"
Nov 03 20:59:43.106764 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:43.106711    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"]
Nov 03 20:59:43.107413 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:43.106994    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s" podUID="79e627d8-c0cf-4c48-b165-1da76f3b7043" containerName="istio-proxy" containerID="cri-o://f0dc6b29393ed8b1744798a331d9b12bd05fb547a691812b9f55babe91a3dbee" gracePeriod=30
Nov 03 20:59:48.398257 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.398192    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:59:48.537621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.530171    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-workload-certs\") pod \"79e627d8-c0cf-4c48-b165-1da76f3b7043\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") "
Nov 03 20:59:48.537621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.530264    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-envoy\") pod \"79e627d8-c0cf-4c48-b165-1da76f3b7043\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") "
Nov 03 20:59:48.537621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.530350    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-workload-socket\") pod \"79e627d8-c0cf-4c48-b165-1da76f3b7043\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") "
Nov 03 20:59:48.537621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.530475    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-token\" (UniqueName: \"kubernetes.io/projected/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-token\") pod \"79e627d8-c0cf-4c48-b165-1da76f3b7043\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") "
Nov 03 20:59:48.537621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.530539    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-data\") pod \"79e627d8-c0cf-4c48-b165-1da76f3b7043\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") "
Nov 03 20:59:48.537621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.530658    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-credential-socket\") pod \"79e627d8-c0cf-4c48-b165-1da76f3b7043\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") "
Nov 03 20:59:48.537621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.530683    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e627d8-c0cf-4c48-b165-1da76f3b7043-kube-api-access-8qkg4\" (UniqueName: \"kubernetes.io/projected/79e627d8-c0cf-4c48-b165-1da76f3b7043-kube-api-access-8qkg4\") pod \"79e627d8-c0cf-4c48-b165-1da76f3b7043\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") "
Nov 03 20:59:48.537621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.530716    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-podinfo\") pod \"79e627d8-c0cf-4c48-b165-1da76f3b7043\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") "
Nov 03 20:59:48.537621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.530753    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79e627d8-c0cf-4c48-b165-1da76f3b7043-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e627d8-c0cf-4c48-b165-1da76f3b7043-istiod-ca-cert\") pod \"79e627d8-c0cf-4c48-b165-1da76f3b7043\" (UID: \"79e627d8-c0cf-4c48-b165-1da76f3b7043\") "
Nov 03 20:59:48.537621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.531859    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e627d8-c0cf-4c48-b165-1da76f3b7043-istiod-ca-cert" pod "79e627d8-c0cf-4c48-b165-1da76f3b7043" (UID: "79e627d8-c0cf-4c48-b165-1da76f3b7043"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:59:48.537621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.532757    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-workload-certs" pod "79e627d8-c0cf-4c48-b165-1da76f3b7043" (UID: "79e627d8-c0cf-4c48-b165-1da76f3b7043"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:48.537621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.534299    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-credential-socket" pod "79e627d8-c0cf-4c48-b165-1da76f3b7043" (UID: "79e627d8-c0cf-4c48-b165-1da76f3b7043"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:48.537621 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.534567    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-workload-socket" pod "79e627d8-c0cf-4c48-b165-1da76f3b7043" (UID: "79e627d8-c0cf-4c48-b165-1da76f3b7043"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:48.545212 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.544925    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-data" pod "79e627d8-c0cf-4c48-b165-1da76f3b7043" (UID: "79e627d8-c0cf-4c48-b165-1da76f3b7043"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:48.551072 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.551041    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-podinfo" pod "79e627d8-c0cf-4c48-b165-1da76f3b7043" (UID: "79e627d8-c0cf-4c48-b165-1da76f3b7043"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:59:48.552010 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.551980    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e627d8-c0cf-4c48-b165-1da76f3b7043-kube-api-access-8qkg4" pod "79e627d8-c0cf-4c48-b165-1da76f3b7043" (UID: "79e627d8-c0cf-4c48-b165-1da76f3b7043"). InnerVolumeSpecName "kube-api-access-8qkg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:48.552452 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.552429    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-token" pod "79e627d8-c0cf-4c48-b165-1da76f3b7043" (UID: "79e627d8-c0cf-4c48-b165-1da76f3b7043"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:48.552560 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.552441    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-envoy" pod "79e627d8-c0cf-4c48-b165-1da76f3b7043" (UID: "79e627d8-c0cf-4c48-b165-1da76f3b7043"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:48.633145 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.633104    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:48.633145 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.633145    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qkg4\" (UniqueName: \"kubernetes.io/projected/79e627d8-c0cf-4c48-b165-1da76f3b7043-kube-api-access-8qkg4\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:48.633145 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.633155    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:48.633402 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.633164    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e627d8-c0cf-4c48-b165-1da76f3b7043-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:48.633402 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.633173    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:48.633402 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.633184    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:48.633402 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.633192    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:48.633402 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.633200    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:48.633402 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.633212    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e627d8-c0cf-4c48-b165-1da76f3b7043-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:48.868724 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.868683    4421 generic.go:356] "Generic (PLEG): container finished" podID="79e627d8-c0cf-4c48-b165-1da76f3b7043" containerID="f0dc6b29393ed8b1744798a331d9b12bd05fb547a691812b9f55babe91a3dbee" exitCode=0
Nov 03 20:59:48.868923 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.868735    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"
Nov 03 20:59:48.868923 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.868776    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s" event={"ID":"79e627d8-c0cf-4c48-b165-1da76f3b7043","Type":"ContainerDied","Data":"f0dc6b29393ed8b1744798a331d9b12bd05fb547a691812b9f55babe91a3dbee"}
Nov 03 20:59:48.868923 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.868831    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s" event={"ID":"79e627d8-c0cf-4c48-b165-1da76f3b7043","Type":"ContainerDied","Data":"9e434d9244331266f90eeb59945875bba44c91eb3d3b33c4f124f5f75f568fb6"}
Nov 03 20:59:48.868923 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.868853    4421 scope.go:122] "RemoveContainer" containerID="f0dc6b29393ed8b1744798a331d9b12bd05fb547a691812b9f55babe91a3dbee"
Nov 03 20:59:48.881456 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.881433    4421 scope.go:122] "RemoveContainer" containerID="f0dc6b29393ed8b1744798a331d9b12bd05fb547a691812b9f55babe91a3dbee"
Nov 03 20:59:48.881822 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:59:48.881793    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0dc6b29393ed8b1744798a331d9b12bd05fb547a691812b9f55babe91a3dbee\": container with ID starting with f0dc6b29393ed8b1744798a331d9b12bd05fb547a691812b9f55babe91a3dbee not found: ID does not exist" containerID="f0dc6b29393ed8b1744798a331d9b12bd05fb547a691812b9f55babe91a3dbee"
Nov 03 20:59:48.881929 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.881835    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0dc6b29393ed8b1744798a331d9b12bd05fb547a691812b9f55babe91a3dbee"} err="failed to get container status \"f0dc6b29393ed8b1744798a331d9b12bd05fb547a691812b9f55babe91a3dbee\": rpc error: code = NotFound desc = could not find container \"f0dc6b29393ed8b1744798a331d9b12bd05fb547a691812b9f55babe91a3dbee\": container with ID starting with f0dc6b29393ed8b1744798a331d9b12bd05fb547a691812b9f55babe91a3dbee not found: ID does not exist"
Nov 03 20:59:48.894472 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.894450    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"]
Nov 03 20:59:48.906876 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:48.906852    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-30e9de-openshift-default-cbd59dfb5-4kx4s"]
Nov 03 20:59:49.631183 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:59:49.631143    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 20:59:50.020652 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:50.020617    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79e627d8-c0cf-4c48-b165-1da76f3b7043" path="/var/lib/kubelet/pods/79e627d8-c0cf-4c48-b165-1da76f3b7043/volumes"
Nov 03 20:59:54.335390 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:54.335355    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j"]
Nov 03 20:59:54.354269 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:54.354232    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p"]
Nov 03 20:59:59.336180 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.336113    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j" podUID="2f5d225a-004c-45c2-944b-987796507d35" containerName="payload-processing" containerID="cri-o://eeaab1516e5bd83d097d73cdb6ea781a4add20e3dad00228943eaa179a3aec20" gracePeriod=25
Nov 03 20:59:59.355845 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.355719    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p" podUID="c1d072eb-7506-40d4-ad37-99b01543157b" containerName="payload-pre-processing" containerID="cri-o://9ab8b35faa70f7c5a3375091c8f80e209f202f2aafc2bc0e0937b0cbbbc15ff2" gracePeriod=25
Nov 03 20:59:59.648883 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.648851    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j"
Nov 03 20:59:59.689352 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.689316    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p"
Nov 03 20:59:59.722031 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.721993    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f5d225a-004c-45c2-944b-987796507d35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5d225a-004c-45c2-944b-987796507d35-plugins-config-volume\") pod \"2f5d225a-004c-45c2-944b-987796507d35\" (UID: \"2f5d225a-004c-45c2-944b-987796507d35\") "
Nov 03 20:59:59.722228 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.722161    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d072eb-7506-40d4-ad37-99b01543157b-kube-api-access-n96m8\" (UniqueName: \"kubernetes.io/projected/c1d072eb-7506-40d4-ad37-99b01543157b-kube-api-access-n96m8\") pod \"c1d072eb-7506-40d4-ad37-99b01543157b\" (UID: \"c1d072eb-7506-40d4-ad37-99b01543157b\") "
Nov 03 20:59:59.722364 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.722344    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1d072eb-7506-40d4-ad37-99b01543157b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1d072eb-7506-40d4-ad37-99b01543157b-plugins-config-volume\") pod \"c1d072eb-7506-40d4-ad37-99b01543157b\" (UID: \"c1d072eb-7506-40d4-ad37-99b01543157b\") "
Nov 03 20:59:59.722465 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.722395    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f5d225a-004c-45c2-944b-987796507d35-kube-api-access-v78l4\" (UniqueName: \"kubernetes.io/projected/2f5d225a-004c-45c2-944b-987796507d35-kube-api-access-v78l4\") pod \"2f5d225a-004c-45c2-944b-987796507d35\" (UID: \"2f5d225a-004c-45c2-944b-987796507d35\") "
Nov 03 20:59:59.722465 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.722401    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f5d225a-004c-45c2-944b-987796507d35-plugins-config-volume" pod "2f5d225a-004c-45c2-944b-987796507d35" (UID: "2f5d225a-004c-45c2-944b-987796507d35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:59:59.723138 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.723110    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5d225a-004c-45c2-944b-987796507d35-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:59.723251 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.722907    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d072eb-7506-40d4-ad37-99b01543157b-plugins-config-volume" pod "c1d072eb-7506-40d4-ad37-99b01543157b" (UID: "c1d072eb-7506-40d4-ad37-99b01543157b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:59:59.724815 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.724790    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d072eb-7506-40d4-ad37-99b01543157b-kube-api-access-n96m8" pod "c1d072eb-7506-40d4-ad37-99b01543157b" (UID: "c1d072eb-7506-40d4-ad37-99b01543157b"). InnerVolumeSpecName "kube-api-access-n96m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:59.725090 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.725067    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f5d225a-004c-45c2-944b-987796507d35-kube-api-access-v78l4" pod "2f5d225a-004c-45c2-944b-987796507d35" (UID: "2f5d225a-004c-45c2-944b-987796507d35"). InnerVolumeSpecName "kube-api-access-v78l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:59.825061 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.825026    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n96m8\" (UniqueName: \"kubernetes.io/projected/c1d072eb-7506-40d4-ad37-99b01543157b-kube-api-access-n96m8\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:59.825061 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.825055    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1d072eb-7506-40d4-ad37-99b01543157b-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:59.825061 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.825065    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v78l4\" (UniqueName: \"kubernetes.io/projected/2f5d225a-004c-45c2-944b-987796507d35-kube-api-access-v78l4\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:59.949134 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.949100    4421 generic.go:356] "Generic (PLEG): container finished" podID="c1d072eb-7506-40d4-ad37-99b01543157b" containerID="9ab8b35faa70f7c5a3375091c8f80e209f202f2aafc2bc0e0937b0cbbbc15ff2" exitCode=0
Nov 03 20:59:59.949325 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.949199    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p"
Nov 03 20:59:59.949325 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.949224    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p" event={"ID":"c1d072eb-7506-40d4-ad37-99b01543157b","Type":"ContainerDied","Data":"9ab8b35faa70f7c5a3375091c8f80e209f202f2aafc2bc0e0937b0cbbbc15ff2"}
Nov 03 20:59:59.949325 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.949276    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p" event={"ID":"c1d072eb-7506-40d4-ad37-99b01543157b","Type":"ContainerDied","Data":"d218371806974215f1c8231efbf731c5f9a78bf81db766bbadcf2d6784bfe800"}
Nov 03 20:59:59.949509 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.949304    4421 scope.go:122] "RemoveContainer" containerID="9ab8b35faa70f7c5a3375091c8f80e209f202f2aafc2bc0e0937b0cbbbc15ff2"
Nov 03 20:59:59.950673 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.950652    4421 generic.go:356] "Generic (PLEG): container finished" podID="2f5d225a-004c-45c2-944b-987796507d35" containerID="eeaab1516e5bd83d097d73cdb6ea781a4add20e3dad00228943eaa179a3aec20" exitCode=0
Nov 03 20:59:59.950789 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.950700    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j"
Nov 03 20:59:59.950789 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.950736    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j" event={"ID":"2f5d225a-004c-45c2-944b-987796507d35","Type":"ContainerDied","Data":"eeaab1516e5bd83d097d73cdb6ea781a4add20e3dad00228943eaa179a3aec20"}
Nov 03 20:59:59.950789 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.950784    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j" event={"ID":"2f5d225a-004c-45c2-944b-987796507d35","Type":"ContainerDied","Data":"7e3b05cea975107549774c600a94a8015ec3ebe28de33fecbfb22728c6f2e749"}
Nov 03 20:59:59.961227 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.961206    4421 scope.go:122] "RemoveContainer" containerID="9ab8b35faa70f7c5a3375091c8f80e209f202f2aafc2bc0e0937b0cbbbc15ff2"
Nov 03 20:59:59.961663 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:59:59.961558    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ab8b35faa70f7c5a3375091c8f80e209f202f2aafc2bc0e0937b0cbbbc15ff2\": container with ID starting with 9ab8b35faa70f7c5a3375091c8f80e209f202f2aafc2bc0e0937b0cbbbc15ff2 not found: ID does not exist" containerID="9ab8b35faa70f7c5a3375091c8f80e209f202f2aafc2bc0e0937b0cbbbc15ff2"
Nov 03 20:59:59.961752 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.961655    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab8b35faa70f7c5a3375091c8f80e209f202f2aafc2bc0e0937b0cbbbc15ff2"} err="failed to get container status \"9ab8b35faa70f7c5a3375091c8f80e209f202f2aafc2bc0e0937b0cbbbc15ff2\": rpc error: code = NotFound desc = could not find container \"9ab8b35faa70f7c5a3375091c8f80e209f202f2aafc2bc0e0937b0cbbbc15ff2\": container with ID starting with 9ab8b35faa70f7c5a3375091c8f80e209f202f2aafc2bc0e0937b0cbbbc15ff2 not found: ID does not exist"
Nov 03 20:59:59.961752 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.961679    4421 scope.go:122] "RemoveContainer" containerID="eeaab1516e5bd83d097d73cdb6ea781a4add20e3dad00228943eaa179a3aec20"
Nov 03 20:59:59.972145 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.972127    4421 scope.go:122] "RemoveContainer" containerID="eeaab1516e5bd83d097d73cdb6ea781a4add20e3dad00228943eaa179a3aec20"
Nov 03 20:59:59.972415 ip-10-0-143-48 kubenswrapper[4421]: E1103 20:59:59.972396    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeaab1516e5bd83d097d73cdb6ea781a4add20e3dad00228943eaa179a3aec20\": container with ID starting with eeaab1516e5bd83d097d73cdb6ea781a4add20e3dad00228943eaa179a3aec20 not found: ID does not exist" containerID="eeaab1516e5bd83d097d73cdb6ea781a4add20e3dad00228943eaa179a3aec20"
Nov 03 20:59:59.972484 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.972425    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeaab1516e5bd83d097d73cdb6ea781a4add20e3dad00228943eaa179a3aec20"} err="failed to get container status \"eeaab1516e5bd83d097d73cdb6ea781a4add20e3dad00228943eaa179a3aec20\": rpc error: code = NotFound desc = could not find container \"eeaab1516e5bd83d097d73cdb6ea781a4add20e3dad00228943eaa179a3aec20\": container with ID starting with eeaab1516e5bd83d097d73cdb6ea781a4add20e3dad00228943eaa179a3aec20 not found: ID does not exist"
Nov 03 20:59:59.977400 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.977374    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p"]
Nov 03 20:59:59.984158 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.984138    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-466f83-5fcfcc9947-m6r7p"]
Nov 03 20:59:59.995212 ip-10-0-143-48 kubenswrapper[4421]: I1103 20:59:59.995190    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j"]
Nov 03 21:00:00.002985 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.002961    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-466f83-79cb4b6b8-5ph2j"]
Nov 03 21:00:00.023376 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.023297    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f5d225a-004c-45c2-944b-987796507d35" path="/var/lib/kubelet/pods/2f5d225a-004c-45c2-944b-987796507d35/volumes"
Nov 03 21:00:00.024180 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.024154    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1d072eb-7506-40d4-ad37-99b01543157b" path="/var/lib/kubelet/pods/c1d072eb-7506-40d4-ad37-99b01543157b/volumes"
Nov 03 21:00:00.221878 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.221794    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp"]
Nov 03 21:00:00.222907 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.222880    4421 cpu_manager.go:414] "containerMap: removing container" podUID="2fcd696c-4d3c-484f-84e7-c6b9b7f05005" containerName="payload-pre-processing"
Nov 03 21:00:00.223100 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223082    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fcd696c-4d3c-484f-84e7-c6b9b7f05005" containerName="payload-pre-processing"
Nov 03 21:00:00.223225 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223212    4421 cpu_manager.go:414] "containerMap: removing container" podUID="a12a350f-f3a6-4f34-b594-a29f2da298f4" containerName="payload-processing"
Nov 03 21:00:00.223319 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223306    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a12a350f-f3a6-4f34-b594-a29f2da298f4" containerName="payload-processing"
Nov 03 21:00:00.223430 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223411    4421 cpu_manager.go:414] "containerMap: removing container" podUID="c1d072eb-7506-40d4-ad37-99b01543157b" containerName="payload-pre-processing"
Nov 03 21:00:00.223530 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223429    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d072eb-7506-40d4-ad37-99b01543157b" containerName="payload-pre-processing"
Nov 03 21:00:00.223530 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223451    4421 cpu_manager.go:414] "containerMap: removing container" podUID="53001e2c-5c08-4ad6-af62-053a80639a3f" containerName="payload-pre-processing"
Nov 03 21:00:00.223530 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223459    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53001e2c-5c08-4ad6-af62-053a80639a3f" containerName="payload-pre-processing"
Nov 03 21:00:00.223530 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223479    4421 cpu_manager.go:414] "containerMap: removing container" podUID="2f5d225a-004c-45c2-944b-987796507d35" containerName="payload-processing"
Nov 03 21:00:00.223530 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223486    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f5d225a-004c-45c2-944b-987796507d35" containerName="payload-processing"
Nov 03 21:00:00.223530 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223503    4421 cpu_manager.go:414] "containerMap: removing container" podUID="9c1a8f22-edce-40d8-8216-adb79f5cd692" containerName="maas-api"
Nov 03 21:00:00.223530 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223511    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1a8f22-edce-40d8-8216-adb79f5cd692" containerName="maas-api"
Nov 03 21:00:00.223530 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223527    4421 cpu_manager.go:414] "containerMap: removing container" podUID="e5fdfb8c-95f4-47c8-8154-4864c5485f24" containerName="payload-processing"
Nov 03 21:00:00.223880 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223534    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5fdfb8c-95f4-47c8-8154-4864c5485f24" containerName="payload-processing"
Nov 03 21:00:00.223880 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223552    4421 cpu_manager.go:414] "containerMap: removing container" podUID="1106b419-0b67-47af-82eb-b2dd73d03f00" containerName="istio-proxy"
Nov 03 21:00:00.223880 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223559    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1106b419-0b67-47af-82eb-b2dd73d03f00" containerName="istio-proxy"
Nov 03 21:00:00.223880 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223589    4421 cpu_manager.go:414] "containerMap: removing container" podUID="79e627d8-c0cf-4c48-b165-1da76f3b7043" containerName="istio-proxy"
Nov 03 21:00:00.223880 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223613    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e627d8-c0cf-4c48-b165-1da76f3b7043" containerName="istio-proxy"
Nov 03 21:00:00.223880 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223626    4421 cpu_manager.go:414] "containerMap: removing container" podUID="0130f392-da22-43d5-87e5-6f69d6475d7d" containerName="maas-api"
Nov 03 21:00:00.223880 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223633    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0130f392-da22-43d5-87e5-6f69d6475d7d" containerName="maas-api"
Nov 03 21:00:00.223880 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223659    4421 cpu_manager.go:414] "containerMap: removing container" podUID="bd5b0764-48b8-4909-b75d-8f4b8575b015" containerName="payload-processing"
Nov 03 21:00:00.223880 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223668    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd5b0764-48b8-4909-b75d-8f4b8575b015" containerName="payload-processing"
Nov 03 21:00:00.223880 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223688    4421 cpu_manager.go:414] "containerMap: removing container" podUID="8fdce1ab-1499-40bf-8f67-f1f23b2959b8" containerName="payload-pre-processing"
Nov 03 21:00:00.223880 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223695    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fdce1ab-1499-40bf-8f67-f1f23b2959b8" containerName="payload-pre-processing"
Nov 03 21:00:00.223880 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223851    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f5d225a-004c-45c2-944b-987796507d35" containerName="payload-processing"
Nov 03 21:00:00.223880 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223869    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1a8f22-edce-40d8-8216-adb79f5cd692" containerName="maas-api"
Nov 03 21:00:00.223880 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223879    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5fdfb8c-95f4-47c8-8154-4864c5485f24" containerName="payload-processing"
Nov 03 21:00:00.224431 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223894    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="79e627d8-c0cf-4c48-b165-1da76f3b7043" containerName="istio-proxy"
Nov 03 21:00:00.224431 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223904    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0130f392-da22-43d5-87e5-6f69d6475d7d" containerName="maas-api"
Nov 03 21:00:00.224431 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223916    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd5b0764-48b8-4909-b75d-8f4b8575b015" containerName="payload-processing"
Nov 03 21:00:00.224431 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223928    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fdce1ab-1499-40bf-8f67-f1f23b2959b8" containerName="payload-pre-processing"
Nov 03 21:00:00.224431 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223939    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fcd696c-4d3c-484f-84e7-c6b9b7f05005" containerName="payload-pre-processing"
Nov 03 21:00:00.224431 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223948    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a12a350f-f3a6-4f34-b594-a29f2da298f4" containerName="payload-processing"
Nov 03 21:00:00.224431 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223957    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d072eb-7506-40d4-ad37-99b01543157b" containerName="payload-pre-processing"
Nov 03 21:00:00.224431 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.223965    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="53001e2c-5c08-4ad6-af62-053a80639a3f" containerName="payload-pre-processing"
Nov 03 21:00:00.228654 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.228632    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp"
Nov 03 21:00:00.241751 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.241721    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp"]
Nov 03 21:00:00.248383 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.248357    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h"]
Nov 03 21:00:00.254068 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.254047    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h"
Nov 03 21:00:00.258782 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.258269    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h"]
Nov 03 21:00:00.266701 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.266655    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\""
Nov 03 21:00:00.275416 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.275386    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz"]
Nov 03 21:00:00.280926 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.280880    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz"]
Nov 03 21:00:00.281069 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.280990    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz"
Nov 03 21:00:00.332803 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.332772    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b472a9-c812-4308-895f-fd5193af34f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h\" (UID: \"68b472a9-c812-4308-895f-fd5193af34f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h"
Nov 03 21:00:00.332953 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.332924    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6fhg\" (UniqueName: \"kubernetes.io/projected/7caf12c0-4e78-4105-bc0c-7a7e8937e3ac-kube-api-access-f6fhg\") pod \"maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz\" (UID: \"7caf12c0-4e78-4105-bc0c-7a7e8937e3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz"
Nov 03 21:00:00.333092 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.333076    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9zqw\" (UniqueName: \"kubernetes.io/projected/629e965e-1f05-4fdd-a1ef-40aa9da9e3a0-kube-api-access-j9zqw\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp\" (UID: \"629e965e-1f05-4fdd-a1ef-40aa9da9e3a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp"
Nov 03 21:00:00.333212 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.333184    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcsxh\" (UniqueName: \"kubernetes.io/projected/68b472a9-c812-4308-895f-fd5193af34f6-kube-api-access-qcsxh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h\" (UID: \"68b472a9-c812-4308-895f-fd5193af34f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h"
Nov 03 21:00:00.333329 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.333224    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/629e965e-1f05-4fdd-a1ef-40aa9da9e3a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp\" (UID: \"629e965e-1f05-4fdd-a1ef-40aa9da9e3a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp"
Nov 03 21:00:00.333329 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.333285    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7caf12c0-4e78-4105-bc0c-7a7e8937e3ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz\" (UID: \"7caf12c0-4e78-4105-bc0c-7a7e8937e3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz"
Nov 03 21:00:00.435069 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.435025    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6fhg\" (UniqueName: \"kubernetes.io/projected/7caf12c0-4e78-4105-bc0c-7a7e8937e3ac-kube-api-access-f6fhg\") pod \"maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz\" (UID: \"7caf12c0-4e78-4105-bc0c-7a7e8937e3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz"
Nov 03 21:00:00.435573 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.435162    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9zqw\" (UniqueName: \"kubernetes.io/projected/629e965e-1f05-4fdd-a1ef-40aa9da9e3a0-kube-api-access-j9zqw\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp\" (UID: \"629e965e-1f05-4fdd-a1ef-40aa9da9e3a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp"
Nov 03 21:00:00.435573 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.435246    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcsxh\" (UniqueName: \"kubernetes.io/projected/68b472a9-c812-4308-895f-fd5193af34f6-kube-api-access-qcsxh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h\" (UID: \"68b472a9-c812-4308-895f-fd5193af34f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h"
Nov 03 21:00:00.435573 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.435278    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/629e965e-1f05-4fdd-a1ef-40aa9da9e3a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp\" (UID: \"629e965e-1f05-4fdd-a1ef-40aa9da9e3a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp"
Nov 03 21:00:00.435573 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.435314    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7caf12c0-4e78-4105-bc0c-7a7e8937e3ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz\" (UID: \"7caf12c0-4e78-4105-bc0c-7a7e8937e3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz"
Nov 03 21:00:00.435573 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.435378    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b472a9-c812-4308-895f-fd5193af34f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h\" (UID: \"68b472a9-c812-4308-895f-fd5193af34f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h"
Nov 03 21:00:00.435941 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.435921    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/629e965e-1f05-4fdd-a1ef-40aa9da9e3a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp\" (UID: \"629e965e-1f05-4fdd-a1ef-40aa9da9e3a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp"
Nov 03 21:00:00.436016 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.435983    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7caf12c0-4e78-4105-bc0c-7a7e8937e3ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz\" (UID: \"7caf12c0-4e78-4105-bc0c-7a7e8937e3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz"
Nov 03 21:00:00.436144 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.436123    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b472a9-c812-4308-895f-fd5193af34f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h\" (UID: \"68b472a9-c812-4308-895f-fd5193af34f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h"
Nov 03 21:00:00.447359 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.447332    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcsxh\" (UniqueName: \"kubernetes.io/projected/68b472a9-c812-4308-895f-fd5193af34f6-kube-api-access-qcsxh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h\" (UID: \"68b472a9-c812-4308-895f-fd5193af34f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h"
Nov 03 21:00:00.447538 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.447514    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6fhg\" (UniqueName: \"kubernetes.io/projected/7caf12c0-4e78-4105-bc0c-7a7e8937e3ac-kube-api-access-f6fhg\") pod \"maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz\" (UID: \"7caf12c0-4e78-4105-bc0c-7a7e8937e3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz"
Nov 03 21:00:00.447587 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.447561    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9zqw\" (UniqueName: \"kubernetes.io/projected/629e965e-1f05-4fdd-a1ef-40aa9da9e3a0-kube-api-access-j9zqw\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp\" (UID: \"629e965e-1f05-4fdd-a1ef-40aa9da9e3a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp"
Nov 03 21:00:00.559110 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.559020    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp"
Nov 03 21:00:00.574930 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.574898    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h"
Nov 03 21:00:00.599264 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.599226    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz"
Nov 03 21:00:00.842975 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.842945    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp"]
Nov 03 21:00:00.849181 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:00:00.848418    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629e965e_1f05_4fdd_a1ef_40aa9da9e3a0.slice/crio-b309e1b1d2fa52e37db5d37f3b287d7a7f195db1a8f82c897095152b7d490156 WatchSource:0}: Error finding container b309e1b1d2fa52e37db5d37f3b287d7a7f195db1a8f82c897095152b7d490156: Status 404 returned error can't find the container with id b309e1b1d2fa52e37db5d37f3b287d7a7f195db1a8f82c897095152b7d490156
Nov 03 21:00:00.870376 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.870349    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h"]
Nov 03 21:00:00.870853 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:00:00.870824    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b472a9_c812_4308_895f_fd5193af34f6.slice/crio-0b8d50791b7a6f1f5ddbfa63b3519af4722deaa0cd948ac9cecec5699195378e WatchSource:0}: Error finding container 0b8d50791b7a6f1f5ddbfa63b3519af4722deaa0cd948ac9cecec5699195378e: Status 404 returned error can't find the container with id 0b8d50791b7a6f1f5ddbfa63b3519af4722deaa0cd948ac9cecec5699195378e
Nov 03 21:00:00.925102 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.925074    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz"]
Nov 03 21:00:00.925857 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:00:00.925828    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7caf12c0_4e78_4105_bc0c_7a7e8937e3ac.slice/crio-f3adada943ca5aeb4615c413b88f1532ab479d76519b43220104ff3802302ebb WatchSource:0}: Error finding container f3adada943ca5aeb4615c413b88f1532ab479d76519b43220104ff3802302ebb: Status 404 returned error can't find the container with id f3adada943ca5aeb4615c413b88f1532ab479d76519b43220104ff3802302ebb
Nov 03 21:00:00.960892 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.960865    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp" event={"ID":"629e965e-1f05-4fdd-a1ef-40aa9da9e3a0","Type":"ContainerStarted","Data":"b309e1b1d2fa52e37db5d37f3b287d7a7f195db1a8f82c897095152b7d490156"}
Nov 03 21:00:00.962127 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.962095    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h" event={"ID":"68b472a9-c812-4308-895f-fd5193af34f6","Type":"ContainerStarted","Data":"0b8d50791b7a6f1f5ddbfa63b3519af4722deaa0cd948ac9cecec5699195378e"}
Nov 03 21:00:00.963361 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:00.963337    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz" event={"ID":"7caf12c0-4e78-4105-bc0c-7a7e8937e3ac","Type":"ContainerStarted","Data":"f3adada943ca5aeb4615c413b88f1532ab479d76519b43220104ff3802302ebb"}
Nov 03 21:00:01.973335 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:01.973305    4421 generic.go:356] "Generic (PLEG): container finished" podID="7caf12c0-4e78-4105-bc0c-7a7e8937e3ac" containerID="712e45776c6f67e0179040c92d7ec831cf4e27ec79f886ad54288f0671c71e29" exitCode=60
Nov 03 21:00:01.973798 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:01.973378    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz" event={"ID":"7caf12c0-4e78-4105-bc0c-7a7e8937e3ac","Type":"ContainerDied","Data":"712e45776c6f67e0179040c92d7ec831cf4e27ec79f886ad54288f0671c71e29"}
Nov 03 21:00:01.973798 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:01.973699    4421 scope.go:122] "RemoveContainer" containerID="712e45776c6f67e0179040c92d7ec831cf4e27ec79f886ad54288f0671c71e29"
Nov 03 21:00:01.975127 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:01.975101    4421 generic.go:356] "Generic (PLEG): container finished" podID="629e965e-1f05-4fdd-a1ef-40aa9da9e3a0" containerID="71a8d6a2aa772d652b9c7d746b479d2608b0121e8774b48764ce7f34a53db153" exitCode=60
Nov 03 21:00:01.975252 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:01.975186    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp" event={"ID":"629e965e-1f05-4fdd-a1ef-40aa9da9e3a0","Type":"ContainerDied","Data":"71a8d6a2aa772d652b9c7d746b479d2608b0121e8774b48764ce7f34a53db153"}
Nov 03 21:00:01.975385 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:01.975367    4421 scope.go:122] "RemoveContainer" containerID="71a8d6a2aa772d652b9c7d746b479d2608b0121e8774b48764ce7f34a53db153"
Nov 03 21:00:01.976988 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:01.976966    4421 generic.go:356] "Generic (PLEG): container finished" podID="68b472a9-c812-4308-895f-fd5193af34f6" containerID="d24670556c0718a1c5e04cbdf6bc5c7f5a48e060530c70db2d04d51f920683f0" exitCode=60
Nov 03 21:00:01.977078 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:01.977012    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h" event={"ID":"68b472a9-c812-4308-895f-fd5193af34f6","Type":"ContainerDied","Data":"d24670556c0718a1c5e04cbdf6bc5c7f5a48e060530c70db2d04d51f920683f0"}
Nov 03 21:00:01.977210 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:01.977197    4421 scope.go:122] "RemoveContainer" containerID="d24670556c0718a1c5e04cbdf6bc5c7f5a48e060530c70db2d04d51f920683f0"
Nov 03 21:00:02.986747 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:02.986717    4421 generic.go:356] "Generic (PLEG): container finished" podID="68b472a9-c812-4308-895f-fd5193af34f6" containerID="239724a2d531ec2278db2575826734b7aa6268cab5883187fe8aea97c29f45be" exitCode=60
Nov 03 21:00:02.987214 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:02.986791    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h" event={"ID":"68b472a9-c812-4308-895f-fd5193af34f6","Type":"ContainerDied","Data":"239724a2d531ec2278db2575826734b7aa6268cab5883187fe8aea97c29f45be"}
Nov 03 21:00:02.987214 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:02.986885    4421 scope.go:122] "RemoveContainer" containerID="d24670556c0718a1c5e04cbdf6bc5c7f5a48e060530c70db2d04d51f920683f0"
Nov 03 21:00:02.987342 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:02.987213    4421 scope.go:122] "RemoveContainer" containerID="239724a2d531ec2278db2575826734b7aa6268cab5883187fe8aea97c29f45be"
Nov 03 21:00:02.987632 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:00:02.987577    4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h_odh-ai-gateway-infra(68b472a9-c812-4308-895f-fd5193af34f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h" podUID="68b472a9-c812-4308-895f-fd5193af34f6"
Nov 03 21:00:02.989116 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:02.989089    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz" event={"ID":"7caf12c0-4e78-4105-bc0c-7a7e8937e3ac","Type":"ContainerStarted","Data":"a346cc13a83a5ab017be3e5267f57985bb5d6d76035e50dd115b04b21fe5187d"}
Nov 03 21:00:02.991231 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:02.991211    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp" event={"ID":"629e965e-1f05-4fdd-a1ef-40aa9da9e3a0","Type":"ContainerStarted","Data":"729c23c205ccef5199ac480ec5d93fe1102061bffc6b856c2e21a8f16c707e09"}
Nov 03 21:00:03.032037 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:03.031993    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz" podStartSLOduration=3.031979376 podStartE2EDuration="3.031979376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:03.030509052 +0000 UTC m=+3533.163637520" watchObservedRunningTime="2026-11-03 21:00:03.031979376 +0000 UTC m=+3533.165107837"
Nov 03 21:00:03.049744 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:03.049684    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp" podStartSLOduration=3.049665579 podStartE2EDuration="3.049665579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:03.04799599 +0000 UTC m=+3533.181124449" watchObservedRunningTime="2026-11-03 21:00:03.049665579 +0000 UTC m=+3533.182794042"
Nov 03 21:00:04.005473 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:04.005440    4421 scope.go:122] "RemoveContainer" containerID="239724a2d531ec2278db2575826734b7aa6268cab5883187fe8aea97c29f45be"
Nov 03 21:00:04.006004 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:00:04.005839    4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h_odh-ai-gateway-infra(68b472a9-c812-4308-895f-fd5193af34f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h" podUID="68b472a9-c812-4308-895f-fd5193af34f6"
Nov 03 21:00:04.010501 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:04.010471    4421 generic.go:356] "Generic (PLEG): container finished" podID="7caf12c0-4e78-4105-bc0c-7a7e8937e3ac" containerID="a346cc13a83a5ab017be3e5267f57985bb5d6d76035e50dd115b04b21fe5187d" exitCode=60
Nov 03 21:00:04.011035 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:04.011016    4421 scope.go:122] "RemoveContainer" containerID="a346cc13a83a5ab017be3e5267f57985bb5d6d76035e50dd115b04b21fe5187d"
Nov 03 21:00:04.011346 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:00:04.011323    4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz_odh-ai-gateway-infra(7caf12c0-4e78-4105-bc0c-7a7e8937e3ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz" podUID="7caf12c0-4e78-4105-bc0c-7a7e8937e3ac"
Nov 03 21:00:04.013980 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:04.013955    4421 generic.go:356] "Generic (PLEG): container finished" podID="629e965e-1f05-4fdd-a1ef-40aa9da9e3a0" containerID="729c23c205ccef5199ac480ec5d93fe1102061bffc6b856c2e21a8f16c707e09" exitCode=60
Nov 03 21:00:04.014304 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:04.014287    4421 scope.go:122] "RemoveContainer" containerID="729c23c205ccef5199ac480ec5d93fe1102061bffc6b856c2e21a8f16c707e09"
Nov 03 21:00:04.014647 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:00:04.014613    4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp_odh-ai-gateway-infra(629e965e-1f05-4fdd-a1ef-40aa9da9e3a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp" podUID="629e965e-1f05-4fdd-a1ef-40aa9da9e3a0"
Nov 03 21:00:04.022164 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:04.022129    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz" event={"ID":"7caf12c0-4e78-4105-bc0c-7a7e8937e3ac","Type":"ContainerDied","Data":"a346cc13a83a5ab017be3e5267f57985bb5d6d76035e50dd115b04b21fe5187d"}
Nov 03 21:00:04.022282 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:04.022186    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp" event={"ID":"629e965e-1f05-4fdd-a1ef-40aa9da9e3a0","Type":"ContainerDied","Data":"729c23c205ccef5199ac480ec5d93fe1102061bffc6b856c2e21a8f16c707e09"}
Nov 03 21:00:04.022282 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:04.022230    4421 scope.go:122] "RemoveContainer" containerID="712e45776c6f67e0179040c92d7ec831cf4e27ec79f886ad54288f0671c71e29"
Nov 03 21:00:04.045902 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:04.045876    4421 scope.go:122] "RemoveContainer" containerID="71a8d6a2aa772d652b9c7d746b479d2608b0121e8774b48764ce7f34a53db153"
Nov 03 21:00:05.023424 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:05.023395    4421 scope.go:122] "RemoveContainer" containerID="729c23c205ccef5199ac480ec5d93fe1102061bffc6b856c2e21a8f16c707e09"
Nov 03 21:00:05.023944 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:00:05.023833    4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp_odh-ai-gateway-infra(629e965e-1f05-4fdd-a1ef-40aa9da9e3a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp" podUID="629e965e-1f05-4fdd-a1ef-40aa9da9e3a0"
Nov 03 21:00:05.025242 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:05.025223    4421 scope.go:122] "RemoveContainer" containerID="a346cc13a83a5ab017be3e5267f57985bb5d6d76035e50dd115b04b21fe5187d"
Nov 03 21:00:05.025535 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:00:05.025517    4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz_odh-ai-gateway-infra(7caf12c0-4e78-4105-bc0c-7a7e8937e3ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz" podUID="7caf12c0-4e78-4105-bc0c-7a7e8937e3ac"
Nov 03 21:00:16.009058 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:16.009030    4421 scope.go:122] "RemoveContainer" containerID="a346cc13a83a5ab017be3e5267f57985bb5d6d76035e50dd115b04b21fe5187d"
Nov 03 21:00:17.009256 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:17.009231    4421 scope.go:122] "RemoveContainer" containerID="239724a2d531ec2278db2575826734b7aa6268cab5883187fe8aea97c29f45be"
Nov 03 21:00:17.009684 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:17.009348    4421 scope.go:122] "RemoveContainer" containerID="729c23c205ccef5199ac480ec5d93fe1102061bffc6b856c2e21a8f16c707e09"
Nov 03 21:00:17.117874 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:17.117842    4421 generic.go:356] "Generic (PLEG): container finished" podID="7caf12c0-4e78-4105-bc0c-7a7e8937e3ac" containerID="6ffa697a9494ed6431dd38a38097bb683ed76cf1af54922e0bf5f3572c78e74b" exitCode=60
Nov 03 21:00:17.118069 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:17.117881    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz" event={"ID":"7caf12c0-4e78-4105-bc0c-7a7e8937e3ac","Type":"ContainerDied","Data":"6ffa697a9494ed6431dd38a38097bb683ed76cf1af54922e0bf5f3572c78e74b"}
Nov 03 21:00:17.118069 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:17.117946    4421 scope.go:122] "RemoveContainer" containerID="a346cc13a83a5ab017be3e5267f57985bb5d6d76035e50dd115b04b21fe5187d"
Nov 03 21:00:17.118488 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:17.118465    4421 scope.go:122] "RemoveContainer" containerID="6ffa697a9494ed6431dd38a38097bb683ed76cf1af54922e0bf5f3572c78e74b"
Nov 03 21:00:17.119045 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:00:17.119021    4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz_odh-ai-gateway-infra(7caf12c0-4e78-4105-bc0c-7a7e8937e3ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz" podUID="7caf12c0-4e78-4105-bc0c-7a7e8937e3ac"
Nov 03 21:00:17.822402 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:17.822372    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"]
Nov 03 21:00:17.827248 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:17.827228    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:17.839465 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:17.836416    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-466f83-openshift-default-dockercfg-6pzmx\""
Nov 03 21:00:17.843636 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:17.841740    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"]
Nov 03 21:00:17.968416 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:17.968374    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-r628m\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:17.968416 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:17.968426    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-r628m\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:17.968732 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:17.968631    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6tlz\" (UniqueName: \"kubernetes.io/projected/bb44d95b-d987-48a4-8220-e1bdf0abfba1-kube-api-access-w6tlz\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-r628m\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:17.968732 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:17.968680    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-r628m\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:17.968851 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:17.968832    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-r628m\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:17.968911 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:17.968862    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-r628m\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:17.968999 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:17.968984    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-r628m\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:17.969047 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:17.969007    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-r628m\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:17.969145 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:17.969129    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-r628m\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:18.070384 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.070355    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-r628m\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:18.070756 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.070405    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-r628m\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:18.070756 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.070428    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-r628m\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:18.070756 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.070498    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6tlz\" (UniqueName: \"kubernetes.io/projected/bb44d95b-d987-48a4-8220-e1bdf0abfba1-kube-api-access-w6tlz\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-r628m\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:18.070756 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.070522    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-r628m\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:18.070756 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.070611    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-r628m\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:18.070756 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.070637    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-r628m\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:18.070756 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.070715    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-r628m\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:18.070756 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.070740    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-r628m\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:18.071091 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.070757    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-r628m\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:18.071091 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.070850    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-r628m\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:18.071091 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.070951    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-r628m\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:18.071209 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.071099    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-r628m\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:18.071209 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.071101    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-r628m\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:18.073014 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.072964    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-r628m\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:18.073232 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.073210    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-r628m\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:18.082738 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.082719    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-r628m\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:18.082922 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.082906    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6tlz\" (UniqueName: \"kubernetes.io/projected/bb44d95b-d987-48a4-8220-e1bdf0abfba1-kube-api-access-w6tlz\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-r628m\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:18.129173 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.129144    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp" event={"ID":"629e965e-1f05-4fdd-a1ef-40aa9da9e3a0","Type":"ContainerStarted","Data":"c195dc9dede8cde0ce4d2c8258541e1f0c7c558825a1bf67019465933afe7450"}
Nov 03 21:00:18.131237 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.131210    4421 generic.go:356] "Generic (PLEG): container finished" podID="68b472a9-c812-4308-895f-fd5193af34f6" containerID="795dffc7e8d8b7918cb8bc10fe36cc658082233cd6e2ea3152eae87b77ddf545" exitCode=60
Nov 03 21:00:18.131359 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.131243    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h" event={"ID":"68b472a9-c812-4308-895f-fd5193af34f6","Type":"ContainerDied","Data":"795dffc7e8d8b7918cb8bc10fe36cc658082233cd6e2ea3152eae87b77ddf545"}
Nov 03 21:00:18.131359 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.131300    4421 scope.go:122] "RemoveContainer" containerID="239724a2d531ec2278db2575826734b7aa6268cab5883187fe8aea97c29f45be"
Nov 03 21:00:18.131725 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.131708    4421 scope.go:122] "RemoveContainer" containerID="795dffc7e8d8b7918cb8bc10fe36cc658082233cd6e2ea3152eae87b77ddf545"
Nov 03 21:00:18.132149 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:00:18.132121    4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h_odh-ai-gateway-infra(68b472a9-c812-4308-895f-fd5193af34f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h" podUID="68b472a9-c812-4308-895f-fd5193af34f6"
Nov 03 21:00:18.143337 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.143314    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:18.155196 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.155168    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz"]
Nov 03 21:00:18.404044 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.404018    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz"
Nov 03 21:00:18.462779 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.462729    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:00:18.462917 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.462858    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:00:18.462917 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.462903    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:00:18.463106 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.463082    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"]
Nov 03 21:00:18.476838 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.476803    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7caf12c0-4e78-4105-bc0c-7a7e8937e3ac-kube-api-access-f6fhg\" (UniqueName: \"kubernetes.io/projected/7caf12c0-4e78-4105-bc0c-7a7e8937e3ac-kube-api-access-f6fhg\") pod \"7caf12c0-4e78-4105-bc0c-7a7e8937e3ac\" (UID: \"7caf12c0-4e78-4105-bc0c-7a7e8937e3ac\") "
Nov 03 21:00:18.477116 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.476897    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7caf12c0-4e78-4105-bc0c-7a7e8937e3ac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7caf12c0-4e78-4105-bc0c-7a7e8937e3ac-maas-api-service-ca\") pod \"7caf12c0-4e78-4105-bc0c-7a7e8937e3ac\" (UID: \"7caf12c0-4e78-4105-bc0c-7a7e8937e3ac\") "
Nov 03 21:00:18.478189 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.478164    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7caf12c0-4e78-4105-bc0c-7a7e8937e3ac-maas-api-service-ca" pod "7caf12c0-4e78-4105-bc0c-7a7e8937e3ac" (UID: "7caf12c0-4e78-4105-bc0c-7a7e8937e3ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:18.488405 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.486325    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7caf12c0-4e78-4105-bc0c-7a7e8937e3ac-kube-api-access-f6fhg" pod "7caf12c0-4e78-4105-bc0c-7a7e8937e3ac" (UID: "7caf12c0-4e78-4105-bc0c-7a7e8937e3ac"). InnerVolumeSpecName "kube-api-access-f6fhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:18.579474 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.579440    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6fhg\" (UniqueName: \"kubernetes.io/projected/7caf12c0-4e78-4105-bc0c-7a7e8937e3ac-kube-api-access-f6fhg\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:18.579474 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:18.579476    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7caf12c0-4e78-4105-bc0c-7a7e8937e3ac-maas-api-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:19.142615 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:19.142561    4421 generic.go:356] "Generic (PLEG): container finished" podID="629e965e-1f05-4fdd-a1ef-40aa9da9e3a0" containerID="c195dc9dede8cde0ce4d2c8258541e1f0c7c558825a1bf67019465933afe7450" exitCode=60
Nov 03 21:00:19.143106 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:19.142640    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp" event={"ID":"629e965e-1f05-4fdd-a1ef-40aa9da9e3a0","Type":"ContainerDied","Data":"c195dc9dede8cde0ce4d2c8258541e1f0c7c558825a1bf67019465933afe7450"}
Nov 03 21:00:19.143106 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:19.142703    4421 scope.go:122] "RemoveContainer" containerID="729c23c205ccef5199ac480ec5d93fe1102061bffc6b856c2e21a8f16c707e09"
Nov 03 21:00:19.143237 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:19.143214    4421 scope.go:122] "RemoveContainer" containerID="c195dc9dede8cde0ce4d2c8258541e1f0c7c558825a1bf67019465933afe7450"
Nov 03 21:00:19.143870 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:00:19.143637    4421 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp_odh-ai-gateway-infra(629e965e-1f05-4fdd-a1ef-40aa9da9e3a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp" podUID="629e965e-1f05-4fdd-a1ef-40aa9da9e3a0"
Nov 03 21:00:19.144747 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:19.144725    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m" event={"ID":"bb44d95b-d987-48a4-8220-e1bdf0abfba1","Type":"ContainerStarted","Data":"2398dce8b38b2679166e1b2548dc9d347bff06b6f9e9816d649f1738ed0bad3e"}
Nov 03 21:00:19.144872 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:19.144771    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m" event={"ID":"bb44d95b-d987-48a4-8220-e1bdf0abfba1","Type":"ContainerStarted","Data":"f197d7c4566de5c7e3980fe3d56df1f663b54e0f46061985dd46190c21408135"}
Nov 03 21:00:19.147886 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:19.147870    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz"
Nov 03 21:00:19.147993 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:19.147902    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz" event={"ID":"7caf12c0-4e78-4105-bc0c-7a7e8937e3ac","Type":"ContainerDied","Data":"f3adada943ca5aeb4615c413b88f1532ab479d76519b43220104ff3802302ebb"}
Nov 03 21:00:19.158413 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:19.158396    4421 scope.go:122] "RemoveContainer" containerID="6ffa697a9494ed6431dd38a38097bb683ed76cf1af54922e0bf5f3572c78e74b"
Nov 03 21:00:19.159777 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:19.159761    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp"]
Nov 03 21:00:19.184180 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:19.184150    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h"]
Nov 03 21:00:19.206292 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:19.206226    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m" podStartSLOduration=2.20621165 podStartE2EDuration="2.20621165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:19.201795029 +0000 UTC m=+3549.334923491" watchObservedRunningTime="2026-11-03 21:00:19.20621165 +0000 UTC m=+3549.339340160"
Nov 03 21:00:19.222184 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:19.222107    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz"]
Nov 03 21:00:19.229696 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:19.229669    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895660-t52kz"]
Nov 03 21:00:19.410847 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:19.410820    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h"
Nov 03 21:00:19.506512 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:19.506434    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68b472a9-c812-4308-895f-fd5193af34f6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b472a9-c812-4308-895f-fd5193af34f6-maas-api-service-ca\") pod \"68b472a9-c812-4308-895f-fd5193af34f6\" (UID: \"68b472a9-c812-4308-895f-fd5193af34f6\") "
Nov 03 21:00:19.506760 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:19.506742    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b472a9-c812-4308-895f-fd5193af34f6-kube-api-access-qcsxh\" (UniqueName: \"kubernetes.io/projected/68b472a9-c812-4308-895f-fd5193af34f6-kube-api-access-qcsxh\") pod \"68b472a9-c812-4308-895f-fd5193af34f6\" (UID: \"68b472a9-c812-4308-895f-fd5193af34f6\") "
Nov 03 21:00:19.506882 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:19.506857    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68b472a9-c812-4308-895f-fd5193af34f6-maas-api-service-ca" pod "68b472a9-c812-4308-895f-fd5193af34f6" (UID: "68b472a9-c812-4308-895f-fd5193af34f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:19.507196 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:19.507184    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b472a9-c812-4308-895f-fd5193af34f6-maas-api-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:19.509016 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:19.508991    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b472a9-c812-4308-895f-fd5193af34f6-kube-api-access-qcsxh" pod "68b472a9-c812-4308-895f-fd5193af34f6" (UID: "68b472a9-c812-4308-895f-fd5193af34f6"). InnerVolumeSpecName "kube-api-access-qcsxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:19.609102 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:19.609018    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcsxh\" (UniqueName: \"kubernetes.io/projected/68b472a9-c812-4308-895f-fd5193af34f6-kube-api-access-qcsxh\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:19.630715 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:00:19.630667    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:00:20.018636 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:20.018591    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7caf12c0-4e78-4105-bc0c-7a7e8937e3ac" path="/var/lib/kubelet/pods/7caf12c0-4e78-4105-bc0c-7a7e8937e3ac/volumes"
Nov 03 21:00:20.152045 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:20.144963    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:20.166947 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:20.166372    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body=
Nov 03 21:00:20.166947 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:20.166456    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m" podUID="bb44d95b-d987-48a4-8220-e1bdf0abfba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused"
Nov 03 21:00:20.177301 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:20.177270    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h" event={"ID":"68b472a9-c812-4308-895f-fd5193af34f6","Type":"ContainerDied","Data":"0b8d50791b7a6f1f5ddbfa63b3519af4722deaa0cd948ac9cecec5699195378e"}
Nov 03 21:00:20.177449 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:20.177302    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h"
Nov 03 21:00:20.177449 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:20.177340    4421 scope.go:122] "RemoveContainer" containerID="795dffc7e8d8b7918cb8bc10fe36cc658082233cd6e2ea3152eae87b77ddf545"
Nov 03 21:00:20.218711 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:20.218675    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h"]
Nov 03 21:00:20.222802 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:20.222773    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895660-59d4h"]
Nov 03 21:00:20.429728 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:20.429700    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp"
Nov 03 21:00:20.533644 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:20.531780    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629e965e-1f05-4fdd-a1ef-40aa9da9e3a0-kube-api-access-j9zqw\" (UniqueName: \"kubernetes.io/projected/629e965e-1f05-4fdd-a1ef-40aa9da9e3a0-kube-api-access-j9zqw\") pod \"629e965e-1f05-4fdd-a1ef-40aa9da9e3a0\" (UID: \"629e965e-1f05-4fdd-a1ef-40aa9da9e3a0\") "
Nov 03 21:00:20.533644 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:20.532044    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/629e965e-1f05-4fdd-a1ef-40aa9da9e3a0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/629e965e-1f05-4fdd-a1ef-40aa9da9e3a0-maas-api-service-ca\") pod \"629e965e-1f05-4fdd-a1ef-40aa9da9e3a0\" (UID: \"629e965e-1f05-4fdd-a1ef-40aa9da9e3a0\") "
Nov 03 21:00:20.533644 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:20.532747    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/629e965e-1f05-4fdd-a1ef-40aa9da9e3a0-maas-api-service-ca" pod "629e965e-1f05-4fdd-a1ef-40aa9da9e3a0" (UID: "629e965e-1f05-4fdd-a1ef-40aa9da9e3a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:20.534719 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:20.534691    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629e965e-1f05-4fdd-a1ef-40aa9da9e3a0-kube-api-access-j9zqw" pod "629e965e-1f05-4fdd-a1ef-40aa9da9e3a0" (UID: "629e965e-1f05-4fdd-a1ef-40aa9da9e3a0"). InnerVolumeSpecName "kube-api-access-j9zqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:20.633805 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:20.633771    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/629e965e-1f05-4fdd-a1ef-40aa9da9e3a0-maas-api-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:20.633805 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:20.633804    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9zqw\" (UniqueName: \"kubernetes.io/projected/629e965e-1f05-4fdd-a1ef-40aa9da9e3a0-kube-api-access-j9zqw\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:21.152623 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:21.146045    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body=
Nov 03 21:00:21.152623 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:21.146117    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m" podUID="bb44d95b-d987-48a4-8220-e1bdf0abfba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused"
Nov 03 21:00:21.193954 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:21.193909    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp" event={"ID":"629e965e-1f05-4fdd-a1ef-40aa9da9e3a0","Type":"ContainerDied","Data":"b309e1b1d2fa52e37db5d37f3b287d7a7f195db1a8f82c897095152b7d490156"}
Nov 03 21:00:21.194162 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:21.194034    4421 scope.go:122] "RemoveContainer" containerID="c195dc9dede8cde0ce4d2c8258541e1f0c7c558825a1bf67019465933afe7450"
Nov 03 21:00:21.194460 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:21.194438    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp"
Nov 03 21:00:21.224640 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:21.224616    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp"]
Nov 03 21:00:21.232904 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:21.232882    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3bcc2b-29895660-kj9jp"]
Nov 03 21:00:22.016585 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:22.016552    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="629e965e-1f05-4fdd-a1ef-40aa9da9e3a0" path="/var/lib/kubelet/pods/629e965e-1f05-4fdd-a1ef-40aa9da9e3a0/volumes"
Nov 03 21:00:22.017057 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:22.017043    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68b472a9-c812-4308-895f-fd5193af34f6" path="/var/lib/kubelet/pods/68b472a9-c812-4308-895f-fd5193af34f6/volumes"
Nov 03 21:00:22.147626 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:22.143969    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body=
Nov 03 21:00:22.147626 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:22.144032    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m" podUID="bb44d95b-d987-48a4-8220-e1bdf0abfba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused"
Nov 03 21:00:23.163633 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:23.162968    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body=
Nov 03 21:00:23.163633 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:23.163030    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m" podUID="bb44d95b-d987-48a4-8220-e1bdf0abfba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused"
Nov 03 21:00:24.146355 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:24.145934    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body=
Nov 03 21:00:24.146355 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:24.145992    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m" podUID="bb44d95b-d987-48a4-8220-e1bdf0abfba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused"
Nov 03 21:00:25.143613 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:25.143566    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body=
Nov 03 21:00:25.144030 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:25.143651    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m" podUID="bb44d95b-d987-48a4-8220-e1bdf0abfba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused"
Nov 03 21:00:26.572878 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:26.572841    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:26.573417 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:26.573389    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:26.574402 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:26.574381    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:38.033087 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:38.033055    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"]
Nov 03 21:00:38.033515 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:38.033276    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m" podUID="bb44d95b-d987-48a4-8220-e1bdf0abfba1" containerName="istio-proxy" containerID="cri-o://2398dce8b38b2679166e1b2548dc9d347bff06b6f9e9816d649f1738ed0bad3e" gracePeriod=30
Nov 03 21:00:41.576149 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:41.576107    4421 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:00:41.576533 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:41.576176    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m" podUID="bb44d95b-d987-48a4-8220-e1bdf0abfba1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:00:43.327564 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.327488    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:43.372389 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.372350    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-data\") pod \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") "
Nov 03 21:00:43.372654 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.372431    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-token\" (UniqueName: \"kubernetes.io/projected/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-token\") pod \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") "
Nov 03 21:00:43.372654 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.372458    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-workload-certs\") pod \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") "
Nov 03 21:00:43.372654 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.372487    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb44d95b-d987-48a4-8220-e1bdf0abfba1-kube-api-access-w6tlz\" (UniqueName: \"kubernetes.io/projected/bb44d95b-d987-48a4-8220-e1bdf0abfba1-kube-api-access-w6tlz\") pod \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") "
Nov 03 21:00:43.372654 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.372525    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-workload-socket\") pod \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") "
Nov 03 21:00:43.372912 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.372681    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-credential-socket\") pod \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") "
Nov 03 21:00:43.372912 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.372743    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-podinfo\") pod \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") "
Nov 03 21:00:43.372912 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.372775    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-envoy\") pod \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") "
Nov 03 21:00:43.372912 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.372863    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istiod-ca-cert\") pod \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\" (UID: \"bb44d95b-d987-48a4-8220-e1bdf0abfba1\") "
Nov 03 21:00:43.373631 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.373582    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istiod-ca-cert" pod "bb44d95b-d987-48a4-8220-e1bdf0abfba1" (UID: "bb44d95b-d987-48a4-8220-e1bdf0abfba1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:43.373876 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.373855    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-data" pod "bb44d95b-d987-48a4-8220-e1bdf0abfba1" (UID: "bb44d95b-d987-48a4-8220-e1bdf0abfba1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:00:43.374414 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.374267    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-workload-socket" pod "bb44d95b-d987-48a4-8220-e1bdf0abfba1" (UID: "bb44d95b-d987-48a4-8220-e1bdf0abfba1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:00:43.375043 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.374768    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-workload-certs" pod "bb44d95b-d987-48a4-8220-e1bdf0abfba1" (UID: "bb44d95b-d987-48a4-8220-e1bdf0abfba1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:00:43.375517 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.375487    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-credential-socket" pod "bb44d95b-d987-48a4-8220-e1bdf0abfba1" (UID: "bb44d95b-d987-48a4-8220-e1bdf0abfba1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:00:43.377112 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.377080    4421 generic.go:356] "Generic (PLEG): container finished" podID="bb44d95b-d987-48a4-8220-e1bdf0abfba1" containerID="2398dce8b38b2679166e1b2548dc9d347bff06b6f9e9816d649f1738ed0bad3e" exitCode=0
Nov 03 21:00:43.377276 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.377256    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m" event={"ID":"bb44d95b-d987-48a4-8220-e1bdf0abfba1","Type":"ContainerDied","Data":"2398dce8b38b2679166e1b2548dc9d347bff06b6f9e9816d649f1738ed0bad3e"}
Nov 03 21:00:43.377363 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.377347    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m" event={"ID":"bb44d95b-d987-48a4-8220-e1bdf0abfba1","Type":"ContainerDied","Data":"f197d7c4566de5c7e3980fe3d56df1f663b54e0f46061985dd46190c21408135"}
Nov 03 21:00:43.377424 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.377387    4421 scope.go:122] "RemoveContainer" containerID="2398dce8b38b2679166e1b2548dc9d347bff06b6f9e9816d649f1738ed0bad3e"
Nov 03 21:00:43.377583 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.377567    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"
Nov 03 21:00:43.378502 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.378323    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-envoy" pod "bb44d95b-d987-48a4-8220-e1bdf0abfba1" (UID: "bb44d95b-d987-48a4-8220-e1bdf0abfba1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:00:43.378502 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.378459    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb44d95b-d987-48a4-8220-e1bdf0abfba1-kube-api-access-w6tlz" pod "bb44d95b-d987-48a4-8220-e1bdf0abfba1" (UID: "bb44d95b-d987-48a4-8220-e1bdf0abfba1"). InnerVolumeSpecName "kube-api-access-w6tlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:43.378952 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.378930    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-token" pod "bb44d95b-d987-48a4-8220-e1bdf0abfba1" (UID: "bb44d95b-d987-48a4-8220-e1bdf0abfba1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:43.379409 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.379124    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-podinfo" pod "bb44d95b-d987-48a4-8220-e1bdf0abfba1" (UID: "bb44d95b-d987-48a4-8220-e1bdf0abfba1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:00:43.411667 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.411640    4421 scope.go:122] "RemoveContainer" containerID="2398dce8b38b2679166e1b2548dc9d347bff06b6f9e9816d649f1738ed0bad3e"
Nov 03 21:00:43.412039 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:00:43.412005    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2398dce8b38b2679166e1b2548dc9d347bff06b6f9e9816d649f1738ed0bad3e\": container with ID starting with 2398dce8b38b2679166e1b2548dc9d347bff06b6f9e9816d649f1738ed0bad3e not found: ID does not exist" containerID="2398dce8b38b2679166e1b2548dc9d347bff06b6f9e9816d649f1738ed0bad3e"
Nov 03 21:00:43.412112 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.412052    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2398dce8b38b2679166e1b2548dc9d347bff06b6f9e9816d649f1738ed0bad3e"} err="failed to get container status \"2398dce8b38b2679166e1b2548dc9d347bff06b6f9e9816d649f1738ed0bad3e\": rpc error: code = NotFound desc = could not find container \"2398dce8b38b2679166e1b2548dc9d347bff06b6f9e9816d649f1738ed0bad3e\": container with ID starting with 2398dce8b38b2679166e1b2548dc9d347bff06b6f9e9816d649f1738ed0bad3e not found: ID does not exist"
Nov 03 21:00:43.474897 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.474864    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:43.474897 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.474893    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:43.474897 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.474902    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:43.475151 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.474911    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:43.475151 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.474921    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6tlz\" (UniqueName: \"kubernetes.io/projected/bb44d95b-d987-48a4-8220-e1bdf0abfba1-kube-api-access-w6tlz\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:43.475151 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.474929    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:43.475151 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.474937    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:43.475151 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.474945    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:43.475151 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.474953    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb44d95b-d987-48a4-8220-e1bdf0abfba1-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:43.705704 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.705674    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"]
Nov 03 21:00:43.712997 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:43.712970    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-r628m"]
Nov 03 21:00:44.016850 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:44.016766    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb44d95b-d987-48a4-8220-e1bdf0abfba1" path="/var/lib/kubelet/pods/bb44d95b-d987-48a4-8220-e1bdf0abfba1/volumes"
Nov 03 21:00:46.907738 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:46.907701    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4"]
Nov 03 21:00:49.630105 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:00:49.630065    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:00:51.909337 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:51.909292    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4" podUID="04160b7a-00d4-47d0-88dd-54646e94e9e5" containerName="payload-processing" containerID="cri-o://e4d4f1e7533e64f8d82e462e21a8b8199cc268a71aba7cd2018e231eefc42ce5" gracePeriod=25
Nov 03 21:00:52.322929 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:52.322869    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4"
Nov 03 21:00:52.359553 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:52.359522    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04160b7a-00d4-47d0-88dd-54646e94e9e5-kube-api-access-7w7bg\" (UniqueName: \"kubernetes.io/projected/04160b7a-00d4-47d0-88dd-54646e94e9e5-kube-api-access-7w7bg\") pod \"04160b7a-00d4-47d0-88dd-54646e94e9e5\" (UID: \"04160b7a-00d4-47d0-88dd-54646e94e9e5\") "
Nov 03 21:00:52.362206 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:52.362181    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04160b7a-00d4-47d0-88dd-54646e94e9e5-kube-api-access-7w7bg" pod "04160b7a-00d4-47d0-88dd-54646e94e9e5" (UID: "04160b7a-00d4-47d0-88dd-54646e94e9e5"). InnerVolumeSpecName "kube-api-access-7w7bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:52.460824 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:52.460726    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04160b7a-00d4-47d0-88dd-54646e94e9e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04160b7a-00d4-47d0-88dd-54646e94e9e5-plugins-config-volume\") pod \"04160b7a-00d4-47d0-88dd-54646e94e9e5\" (UID: \"04160b7a-00d4-47d0-88dd-54646e94e9e5\") "
Nov 03 21:00:52.461640 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:52.461588    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04160b7a-00d4-47d0-88dd-54646e94e9e5-plugins-config-volume" pod "04160b7a-00d4-47d0-88dd-54646e94e9e5" (UID: "04160b7a-00d4-47d0-88dd-54646e94e9e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:52.461874 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:52.461853    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04160b7a-00d4-47d0-88dd-54646e94e9e5-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:52.461985 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:52.461881    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w7bg\" (UniqueName: \"kubernetes.io/projected/04160b7a-00d4-47d0-88dd-54646e94e9e5-kube-api-access-7w7bg\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:52.463665 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:52.463632    4421 generic.go:356] "Generic (PLEG): container finished" podID="04160b7a-00d4-47d0-88dd-54646e94e9e5" containerID="e4d4f1e7533e64f8d82e462e21a8b8199cc268a71aba7cd2018e231eefc42ce5" exitCode=0
Nov 03 21:00:52.463865 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:52.463846    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4" event={"ID":"04160b7a-00d4-47d0-88dd-54646e94e9e5","Type":"ContainerDied","Data":"e4d4f1e7533e64f8d82e462e21a8b8199cc268a71aba7cd2018e231eefc42ce5"}
Nov 03 21:00:52.463978 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:52.463904    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4" event={"ID":"04160b7a-00d4-47d0-88dd-54646e94e9e5","Type":"ContainerDied","Data":"9269017a99b29e6d3ea0e35c347202d5a0903c38ba9e6eafdad71f86cda2c7e5"}
Nov 03 21:00:52.463978 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:52.463854    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4"
Nov 03 21:00:52.464102 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:52.463932    4421 scope.go:122] "RemoveContainer" containerID="e4d4f1e7533e64f8d82e462e21a8b8199cc268a71aba7cd2018e231eefc42ce5"
Nov 03 21:00:52.478456 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:52.478424    4421 scope.go:122] "RemoveContainer" containerID="e4d4f1e7533e64f8d82e462e21a8b8199cc268a71aba7cd2018e231eefc42ce5"
Nov 03 21:00:52.478766 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:00:52.478746    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4d4f1e7533e64f8d82e462e21a8b8199cc268a71aba7cd2018e231eefc42ce5\": container with ID starting with e4d4f1e7533e64f8d82e462e21a8b8199cc268a71aba7cd2018e231eefc42ce5 not found: ID does not exist" containerID="e4d4f1e7533e64f8d82e462e21a8b8199cc268a71aba7cd2018e231eefc42ce5"
Nov 03 21:00:52.478857 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:52.478774    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4d4f1e7533e64f8d82e462e21a8b8199cc268a71aba7cd2018e231eefc42ce5"} err="failed to get container status \"e4d4f1e7533e64f8d82e462e21a8b8199cc268a71aba7cd2018e231eefc42ce5\": rpc error: code = NotFound desc = could not find container \"e4d4f1e7533e64f8d82e462e21a8b8199cc268a71aba7cd2018e231eefc42ce5\": container with ID starting with e4d4f1e7533e64f8d82e462e21a8b8199cc268a71aba7cd2018e231eefc42ce5 not found: ID does not exist"
Nov 03 21:00:52.490783 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:52.490760    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4"]
Nov 03 21:00:52.496844 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:52.496818    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-dce009-5fbcf89498-c29n4"]
Nov 03 21:00:54.017634 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:54.017583    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04160b7a-00d4-47d0-88dd-54646e94e9e5" path="/var/lib/kubelet/pods/04160b7a-00d4-47d0-88dd-54646e94e9e5/volumes"
Nov 03 21:00:55.911612 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:55.907392    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"]
Nov 03 21:00:55.911612 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:00:55.907696    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j" podUID="0a2fda22-a9be-4bf5-8b2b-0e994221c92b" containerName="istio-proxy" containerID="cri-o://fc80028a097b5022a46019d0e712125499641aa2849f75f8367e210a39a7c4ba" gracePeriod=30
Nov 03 21:01:00.041460 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:00.041425    4421 patch_prober.go:29] interesting pod/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:01:00.042129 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:00.042079    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j" podUID="0a2fda22-a9be-4bf5-8b2b-0e994221c92b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:01:01.204411 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.204387    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 21:01:01.321776 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.321742    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-data\") pod \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") "
Nov 03 21:01:01.321949 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.321921    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-kube-api-access-p2dk4\" (UniqueName: \"kubernetes.io/projected/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-kube-api-access-p2dk4\") pod \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") "
Nov 03 21:01:01.322004 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.321969    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-podinfo\") pod \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") "
Nov 03 21:01:01.322064 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.322005    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-data" pod "0a2fda22-a9be-4bf5-8b2b-0e994221c92b" (UID: "0a2fda22-a9be-4bf5-8b2b-0e994221c92b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:01.322064 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.322053    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-token\" (UniqueName: \"kubernetes.io/projected/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-token\") pod \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") "
Nov 03 21:01:01.322182 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.322105    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-workload-certs\") pod \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") "
Nov 03 21:01:01.322241 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.322221    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-workload-socket\") pod \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") "
Nov 03 21:01:01.322298 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.322268    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-credential-socket\") pod \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") "
Nov 03 21:01:01.322577 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.322557    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-envoy\") pod \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") "
Nov 03 21:01:01.322750 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.322345    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-workload-certs" pod "0a2fda22-a9be-4bf5-8b2b-0e994221c92b" (UID: "0a2fda22-a9be-4bf5-8b2b-0e994221c92b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:01.322750 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.322486    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-workload-socket" pod "0a2fda22-a9be-4bf5-8b2b-0e994221c92b" (UID: "0a2fda22-a9be-4bf5-8b2b-0e994221c92b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:01.322750 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.322498    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-credential-socket" pod "0a2fda22-a9be-4bf5-8b2b-0e994221c92b" (UID: "0a2fda22-a9be-4bf5-8b2b-0e994221c92b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:01.322750 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.322642    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istiod-ca-cert\") pod \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\" (UID: \"0a2fda22-a9be-4bf5-8b2b-0e994221c92b\") "
Nov 03 21:01:01.323141 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.323120    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istiod-ca-cert" pod "0a2fda22-a9be-4bf5-8b2b-0e994221c92b" (UID: "0a2fda22-a9be-4bf5-8b2b-0e994221c92b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:01:01.323295 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.323273    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:01.323423 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.323297    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:01.323423 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.323311    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:01.323423 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.323323    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:01.323423 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.323335    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:01.324777 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.324751    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-podinfo" pod "0a2fda22-a9be-4bf5-8b2b-0e994221c92b" (UID: "0a2fda22-a9be-4bf5-8b2b-0e994221c92b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:01:01.325094 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.325069    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-token" pod "0a2fda22-a9be-4bf5-8b2b-0e994221c92b" (UID: "0a2fda22-a9be-4bf5-8b2b-0e994221c92b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:01:01.325189 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.325166    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-kube-api-access-p2dk4" pod "0a2fda22-a9be-4bf5-8b2b-0e994221c92b" (UID: "0a2fda22-a9be-4bf5-8b2b-0e994221c92b"). InnerVolumeSpecName "kube-api-access-p2dk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:01:01.325280 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.325266    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-envoy" pod "0a2fda22-a9be-4bf5-8b2b-0e994221c92b" (UID: "0a2fda22-a9be-4bf5-8b2b-0e994221c92b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:01.424966 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.424935    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2dk4\" (UniqueName: \"kubernetes.io/projected/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-kube-api-access-p2dk4\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:01.424966 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.424960    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:01.424966 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.424970    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:01.425193 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.424978    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a2fda22-a9be-4bf5-8b2b-0e994221c92b-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:01.554480 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.554446    4421 generic.go:356] "Generic (PLEG): container finished" podID="0a2fda22-a9be-4bf5-8b2b-0e994221c92b" containerID="fc80028a097b5022a46019d0e712125499641aa2849f75f8367e210a39a7c4ba" exitCode=0
Nov 03 21:01:01.554707 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.554509    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"
Nov 03 21:01:01.554707 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.554538    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j" event={"ID":"0a2fda22-a9be-4bf5-8b2b-0e994221c92b","Type":"ContainerDied","Data":"fc80028a097b5022a46019d0e712125499641aa2849f75f8367e210a39a7c4ba"}
Nov 03 21:01:01.554707 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.554631    4421 scope.go:122] "RemoveContainer" containerID="fc80028a097b5022a46019d0e712125499641aa2849f75f8367e210a39a7c4ba"
Nov 03 21:01:01.554862 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.554759    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j" event={"ID":"0a2fda22-a9be-4bf5-8b2b-0e994221c92b","Type":"ContainerDied","Data":"a95ee4028f333c52a8e46d072e0a5b4de903e8773215d43021199dd5073e8fd1"}
Nov 03 21:01:01.565994 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.565976    4421 scope.go:122] "RemoveContainer" containerID="6138acb76efaa00253d360733024c7f31ec60285410bbb53cd396b7f98c0917a"
Nov 03 21:01:01.575476 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.575459    4421 scope.go:122] "RemoveContainer" containerID="fc80028a097b5022a46019d0e712125499641aa2849f75f8367e210a39a7c4ba"
Nov 03 21:01:01.575747 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:01:01.575725    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc80028a097b5022a46019d0e712125499641aa2849f75f8367e210a39a7c4ba\": container with ID starting with fc80028a097b5022a46019d0e712125499641aa2849f75f8367e210a39a7c4ba not found: ID does not exist" containerID="fc80028a097b5022a46019d0e712125499641aa2849f75f8367e210a39a7c4ba"
Nov 03 21:01:01.575810 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.575755    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc80028a097b5022a46019d0e712125499641aa2849f75f8367e210a39a7c4ba"} err="failed to get container status \"fc80028a097b5022a46019d0e712125499641aa2849f75f8367e210a39a7c4ba\": rpc error: code = NotFound desc = could not find container \"fc80028a097b5022a46019d0e712125499641aa2849f75f8367e210a39a7c4ba\": container with ID starting with fc80028a097b5022a46019d0e712125499641aa2849f75f8367e210a39a7c4ba not found: ID does not exist"
Nov 03 21:01:01.575810 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.575772    4421 scope.go:122] "RemoveContainer" containerID="6138acb76efaa00253d360733024c7f31ec60285410bbb53cd396b7f98c0917a"
Nov 03 21:01:01.577110 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:01:01.576028    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6138acb76efaa00253d360733024c7f31ec60285410bbb53cd396b7f98c0917a\": container with ID starting with 6138acb76efaa00253d360733024c7f31ec60285410bbb53cd396b7f98c0917a not found: ID does not exist" containerID="6138acb76efaa00253d360733024c7f31ec60285410bbb53cd396b7f98c0917a"
Nov 03 21:01:01.577110 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.576061    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6138acb76efaa00253d360733024c7f31ec60285410bbb53cd396b7f98c0917a"} err="failed to get container status \"6138acb76efaa00253d360733024c7f31ec60285410bbb53cd396b7f98c0917a\": rpc error: code = NotFound desc = could not find container \"6138acb76efaa00253d360733024c7f31ec60285410bbb53cd396b7f98c0917a\": container with ID starting with 6138acb76efaa00253d360733024c7f31ec60285410bbb53cd396b7f98c0917a not found: ID does not exist"
Nov 03 21:01:01.580781 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.580734    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"]
Nov 03 21:01:01.586003 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:01.585981    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-dce009-openshift-default-57db6cbb9c-dpz2j"]
Nov 03 21:01:02.016938 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:02.016909    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a2fda22-a9be-4bf5-8b2b-0e994221c92b" path="/var/lib/kubelet/pods/0a2fda22-a9be-4bf5-8b2b-0e994221c92b/volumes"
Nov 03 21:01:14.341358 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:14.341327    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs"]
Nov 03 21:01:14.343426 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:14.343374    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs" podUID="eabfcd88-38cc-44ae-bb3c-df42c13f0684" containerName="maas-api" containerID="cri-o://64e286545e35b9588f2b45781c356198ed4bfb901e82b9e9e393cab0a7d08c3a" gracePeriod=30
Nov 03 21:01:14.702798 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:14.702732    4421 generic.go:356] "Generic (PLEG): container finished" podID="eabfcd88-38cc-44ae-bb3c-df42c13f0684" containerID="64e286545e35b9588f2b45781c356198ed4bfb901e82b9e9e393cab0a7d08c3a" exitCode=0
Nov 03 21:01:14.702967 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:14.702799    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs" event={"ID":"eabfcd88-38cc-44ae-bb3c-df42c13f0684","Type":"ContainerDied","Data":"64e286545e35b9588f2b45781c356198ed4bfb901e82b9e9e393cab0a7d08c3a"}
Nov 03 21:01:14.719839 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:14.719810    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs"
Nov 03 21:01:14.762868 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:14.762829    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eabfcd88-38cc-44ae-bb3c-df42c13f0684-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabfcd88-38cc-44ae-bb3c-df42c13f0684-openshift-service-ca\") pod \"eabfcd88-38cc-44ae-bb3c-df42c13f0684\" (UID: \"eabfcd88-38cc-44ae-bb3c-df42c13f0684\") "
Nov 03 21:01:14.763071 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:14.762991    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eabfcd88-38cc-44ae-bb3c-df42c13f0684-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eabfcd88-38cc-44ae-bb3c-df42c13f0684-maas-api-tls\") pod \"eabfcd88-38cc-44ae-bb3c-df42c13f0684\" (UID: \"eabfcd88-38cc-44ae-bb3c-df42c13f0684\") "
Nov 03 21:01:14.763071 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:14.763036    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eabfcd88-38cc-44ae-bb3c-df42c13f0684-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabfcd88-38cc-44ae-bb3c-df42c13f0684-metrics-certs\") pod \"eabfcd88-38cc-44ae-bb3c-df42c13f0684\" (UID: \"eabfcd88-38cc-44ae-bb3c-df42c13f0684\") "
Nov 03 21:01:14.763212 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:14.763135    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eabfcd88-38cc-44ae-bb3c-df42c13f0684-kube-api-access-rz9b5\" (UniqueName: \"kubernetes.io/projected/eabfcd88-38cc-44ae-bb3c-df42c13f0684-kube-api-access-rz9b5\") pod \"eabfcd88-38cc-44ae-bb3c-df42c13f0684\" (UID: \"eabfcd88-38cc-44ae-bb3c-df42c13f0684\") "
Nov 03 21:01:14.763268 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:14.763229    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eabfcd88-38cc-44ae-bb3c-df42c13f0684-openshift-service-ca" pod "eabfcd88-38cc-44ae-bb3c-df42c13f0684" (UID: "eabfcd88-38cc-44ae-bb3c-df42c13f0684"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:01:14.763779 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:14.763750    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabfcd88-38cc-44ae-bb3c-df42c13f0684-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:14.766007 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:14.765978    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eabfcd88-38cc-44ae-bb3c-df42c13f0684-kube-api-access-rz9b5" pod "eabfcd88-38cc-44ae-bb3c-df42c13f0684" (UID: "eabfcd88-38cc-44ae-bb3c-df42c13f0684"). InnerVolumeSpecName "kube-api-access-rz9b5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:01:14.766125 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:14.766058    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eabfcd88-38cc-44ae-bb3c-df42c13f0684-metrics-certs" pod "eabfcd88-38cc-44ae-bb3c-df42c13f0684" (UID: "eabfcd88-38cc-44ae-bb3c-df42c13f0684"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:01:14.766206 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:14.766189    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eabfcd88-38cc-44ae-bb3c-df42c13f0684-maas-api-tls" pod "eabfcd88-38cc-44ae-bb3c-df42c13f0684" (UID: "eabfcd88-38cc-44ae-bb3c-df42c13f0684"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:01:14.877091 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:14.877049    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eabfcd88-38cc-44ae-bb3c-df42c13f0684-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:14.877966 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:14.877936    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabfcd88-38cc-44ae-bb3c-df42c13f0684-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:14.878123 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:14.877968    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz9b5\" (UniqueName: \"kubernetes.io/projected/eabfcd88-38cc-44ae-bb3c-df42c13f0684-kube-api-access-rz9b5\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:15.711671 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:15.711643    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs"
Nov 03 21:01:15.712130 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:15.711641    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs" event={"ID":"eabfcd88-38cc-44ae-bb3c-df42c13f0684","Type":"ContainerDied","Data":"fc880127a5b1bed6f4d8d228d771c14d06416f0f14c0cbb2ed8c89e6f4165f44"}
Nov 03 21:01:15.712130 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:15.711768    4421 scope.go:122] "RemoveContainer" containerID="64e286545e35b9588f2b45781c356198ed4bfb901e82b9e9e393cab0a7d08c3a"
Nov 03 21:01:15.736481 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:15.736425    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs"]
Nov 03 21:01:15.740037 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:15.740012    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-05473b-5b45dc6f68-blcvs"]
Nov 03 21:01:16.017274 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:16.017195    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eabfcd88-38cc-44ae-bb3c-df42c13f0684" path="/var/lib/kubelet/pods/eabfcd88-38cc-44ae-bb3c-df42c13f0684/volumes"
Nov 03 21:01:19.628653 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:01:19.628586    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:01:25.140550 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:25.140516    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb"]
Nov 03 21:01:25.150789 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:25.150748    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc"]
Nov 03 21:01:30.141417 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.141364    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb" podUID="c70b0ee2-5d82-4575-86ee-a7adc291b6ff" containerName="payload-processing" containerID="cri-o://5f6621dfeee05b538fe82a1c85388fd9525c729f11f906950f14e417232feea5" gracePeriod=25
Nov 03 21:01:30.151007 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.150965    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc" podUID="0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab" containerName="payload-pre-processing" containerID="cri-o://b8f37ef2fa2157a640af667dbf722552dab3602b37e5e3b58b6e5814ae53dba7" gracePeriod=25
Nov 03 21:01:30.497529 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.497490    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb"
Nov 03 21:01:30.500944 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.500919    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc"
Nov 03 21:01:30.592562 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.592532    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c70b0ee2-5d82-4575-86ee-a7adc291b6ff-kube-api-access-jkhq7\" (UniqueName: \"kubernetes.io/projected/c70b0ee2-5d82-4575-86ee-a7adc291b6ff-kube-api-access-jkhq7\") pod \"c70b0ee2-5d82-4575-86ee-a7adc291b6ff\" (UID: \"c70b0ee2-5d82-4575-86ee-a7adc291b6ff\") "
Nov 03 21:01:30.592777 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.592665    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab-plugins-config-volume\") pod \"0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab\" (UID: \"0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab\") "
Nov 03 21:01:30.592777 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.592712    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c70b0ee2-5d82-4575-86ee-a7adc291b6ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c70b0ee2-5d82-4575-86ee-a7adc291b6ff-plugins-config-volume\") pod \"c70b0ee2-5d82-4575-86ee-a7adc291b6ff\" (UID: \"c70b0ee2-5d82-4575-86ee-a7adc291b6ff\") "
Nov 03 21:01:30.592777 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.592743    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab-kube-api-access-gjmxm\" (UniqueName: \"kubernetes.io/projected/0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab-kube-api-access-gjmxm\") pod \"0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab\" (UID: \"0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab\") "
Nov 03 21:01:30.593318 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.593109    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab-plugins-config-volume" pod "0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab" (UID: "0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:01:30.593318 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.593149    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c70b0ee2-5d82-4575-86ee-a7adc291b6ff-plugins-config-volume" pod "c70b0ee2-5d82-4575-86ee-a7adc291b6ff" (UID: "c70b0ee2-5d82-4575-86ee-a7adc291b6ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:01:30.593802 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.593537    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:30.593802 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.593563    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c70b0ee2-5d82-4575-86ee-a7adc291b6ff-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:30.595336 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.595305    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70b0ee2-5d82-4575-86ee-a7adc291b6ff-kube-api-access-jkhq7" pod "c70b0ee2-5d82-4575-86ee-a7adc291b6ff" (UID: "c70b0ee2-5d82-4575-86ee-a7adc291b6ff"). InnerVolumeSpecName "kube-api-access-jkhq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:01:30.595803 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.595780    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab-kube-api-access-gjmxm" pod "0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab" (UID: "0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab"). InnerVolumeSpecName "kube-api-access-gjmxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:01:30.694648 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.694589    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjmxm\" (UniqueName: \"kubernetes.io/projected/0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab-kube-api-access-gjmxm\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:30.694648 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.694638    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkhq7\" (UniqueName: \"kubernetes.io/projected/c70b0ee2-5d82-4575-86ee-a7adc291b6ff-kube-api-access-jkhq7\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:30.836061 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.835982    4421 generic.go:356] "Generic (PLEG): container finished" podID="c70b0ee2-5d82-4575-86ee-a7adc291b6ff" containerID="5f6621dfeee05b538fe82a1c85388fd9525c729f11f906950f14e417232feea5" exitCode=0
Nov 03 21:01:30.836061 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.836027    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb"
Nov 03 21:01:30.836314 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.836066    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb" event={"ID":"c70b0ee2-5d82-4575-86ee-a7adc291b6ff","Type":"ContainerDied","Data":"5f6621dfeee05b538fe82a1c85388fd9525c729f11f906950f14e417232feea5"}
Nov 03 21:01:30.836314 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.836125    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb" event={"ID":"c70b0ee2-5d82-4575-86ee-a7adc291b6ff","Type":"ContainerDied","Data":"e6ce31a2f936666647ef7e999024a613bbfcda3430878278426328755ab0dd30"}
Nov 03 21:01:30.836314 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.836157    4421 scope.go:122] "RemoveContainer" containerID="5f6621dfeee05b538fe82a1c85388fd9525c729f11f906950f14e417232feea5"
Nov 03 21:01:30.838093 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.838067    4421 generic.go:356] "Generic (PLEG): container finished" podID="0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab" containerID="b8f37ef2fa2157a640af667dbf722552dab3602b37e5e3b58b6e5814ae53dba7" exitCode=0
Nov 03 21:01:30.838208 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.838173    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc"
Nov 03 21:01:30.838208 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.838195    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc" event={"ID":"0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab","Type":"ContainerDied","Data":"b8f37ef2fa2157a640af667dbf722552dab3602b37e5e3b58b6e5814ae53dba7"}
Nov 03 21:01:30.838285 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.838236    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc" event={"ID":"0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab","Type":"ContainerDied","Data":"d3f0c2e236bc8b5e92c72caa7096338e26a60a7309c3786b1ebf2f37bd230004"}
Nov 03 21:01:30.848384 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.848359    4421 scope.go:122] "RemoveContainer" containerID="5f6621dfeee05b538fe82a1c85388fd9525c729f11f906950f14e417232feea5"
Nov 03 21:01:30.848688 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:01:30.848663    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f6621dfeee05b538fe82a1c85388fd9525c729f11f906950f14e417232feea5\": container with ID starting with 5f6621dfeee05b538fe82a1c85388fd9525c729f11f906950f14e417232feea5 not found: ID does not exist" containerID="5f6621dfeee05b538fe82a1c85388fd9525c729f11f906950f14e417232feea5"
Nov 03 21:01:30.848807 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.848698    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f6621dfeee05b538fe82a1c85388fd9525c729f11f906950f14e417232feea5"} err="failed to get container status \"5f6621dfeee05b538fe82a1c85388fd9525c729f11f906950f14e417232feea5\": rpc error: code = NotFound desc = could not find container \"5f6621dfeee05b538fe82a1c85388fd9525c729f11f906950f14e417232feea5\": container with ID starting with 5f6621dfeee05b538fe82a1c85388fd9525c729f11f906950f14e417232feea5 not found: ID does not exist"
Nov 03 21:01:30.848807 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.848721    4421 scope.go:122] "RemoveContainer" containerID="b8f37ef2fa2157a640af667dbf722552dab3602b37e5e3b58b6e5814ae53dba7"
Nov 03 21:01:30.860409 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.860392    4421 scope.go:122] "RemoveContainer" containerID="b8f37ef2fa2157a640af667dbf722552dab3602b37e5e3b58b6e5814ae53dba7"
Nov 03 21:01:30.860686 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:01:30.860664    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8f37ef2fa2157a640af667dbf722552dab3602b37e5e3b58b6e5814ae53dba7\": container with ID starting with b8f37ef2fa2157a640af667dbf722552dab3602b37e5e3b58b6e5814ae53dba7 not found: ID does not exist" containerID="b8f37ef2fa2157a640af667dbf722552dab3602b37e5e3b58b6e5814ae53dba7"
Nov 03 21:01:30.860775 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.860693    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8f37ef2fa2157a640af667dbf722552dab3602b37e5e3b58b6e5814ae53dba7"} err="failed to get container status \"b8f37ef2fa2157a640af667dbf722552dab3602b37e5e3b58b6e5814ae53dba7\": rpc error: code = NotFound desc = could not find container \"b8f37ef2fa2157a640af667dbf722552dab3602b37e5e3b58b6e5814ae53dba7\": container with ID starting with b8f37ef2fa2157a640af667dbf722552dab3602b37e5e3b58b6e5814ae53dba7 not found: ID does not exist"
Nov 03 21:01:30.954755 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.954717    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb"]
Nov 03 21:01:30.976726 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.976699    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-7a92d4-d6d6b6c84-w9rlb"]
Nov 03 21:01:30.994835 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:30.994802    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc"]
Nov 03 21:01:31.005128 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:31.005097    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7a92d4-84cccb45f5-5bgqc"]
Nov 03 21:01:32.019951 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:32.019910    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab" path="/var/lib/kubelet/pods/0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab/volumes"
Nov 03 21:01:32.020561 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:32.020541    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c70b0ee2-5d82-4575-86ee-a7adc291b6ff" path="/var/lib/kubelet/pods/c70b0ee2-5d82-4575-86ee-a7adc291b6ff/volumes"
Nov 03 21:01:36.632144 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:36.632097    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq"]
Nov 03 21:01:36.632706 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:36.632362    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq" podUID="33274f28-f2d3-40fc-a16b-75e90ece007a" containerName="maas-api" containerID="cri-o://56227fd565e459ced9ed09f25db9fd23e1542adbf202fd10c647f90c1566c64f" gracePeriod=30
Nov 03 21:01:36.984689 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:36.973480    4421 generic.go:356] "Generic (PLEG): container finished" podID="33274f28-f2d3-40fc-a16b-75e90ece007a" containerID="56227fd565e459ced9ed09f25db9fd23e1542adbf202fd10c647f90c1566c64f" exitCode=0
Nov 03 21:01:36.984689 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:36.973527    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq" event={"ID":"33274f28-f2d3-40fc-a16b-75e90ece007a","Type":"ContainerDied","Data":"56227fd565e459ced9ed09f25db9fd23e1542adbf202fd10c647f90c1566c64f"}
Nov 03 21:01:37.043561 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:37.043537    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq"
Nov 03 21:01:37.119300 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:37.119268    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33274f28-f2d3-40fc-a16b-75e90ece007a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33274f28-f2d3-40fc-a16b-75e90ece007a-openshift-service-ca\") pod \"33274f28-f2d3-40fc-a16b-75e90ece007a\" (UID: \"33274f28-f2d3-40fc-a16b-75e90ece007a\") "
Nov 03 21:01:37.119486 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:37.119351    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33274f28-f2d3-40fc-a16b-75e90ece007a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/33274f28-f2d3-40fc-a16b-75e90ece007a-metrics-certs\") pod \"33274f28-f2d3-40fc-a16b-75e90ece007a\" (UID: \"33274f28-f2d3-40fc-a16b-75e90ece007a\") "
Nov 03 21:01:37.119486 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:37.119371    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33274f28-f2d3-40fc-a16b-75e90ece007a-kube-api-access-5hnwv\" (UniqueName: \"kubernetes.io/projected/33274f28-f2d3-40fc-a16b-75e90ece007a-kube-api-access-5hnwv\") pod \"33274f28-f2d3-40fc-a16b-75e90ece007a\" (UID: \"33274f28-f2d3-40fc-a16b-75e90ece007a\") "
Nov 03 21:01:37.119714 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:37.119688    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33274f28-f2d3-40fc-a16b-75e90ece007a-openshift-service-ca" pod "33274f28-f2d3-40fc-a16b-75e90ece007a" (UID: "33274f28-f2d3-40fc-a16b-75e90ece007a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:01:37.119960 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:37.119944    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33274f28-f2d3-40fc-a16b-75e90ece007a-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:37.121723 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:37.121699    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33274f28-f2d3-40fc-a16b-75e90ece007a-kube-api-access-5hnwv" pod "33274f28-f2d3-40fc-a16b-75e90ece007a" (UID: "33274f28-f2d3-40fc-a16b-75e90ece007a"). InnerVolumeSpecName "kube-api-access-5hnwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:01:37.121835 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:37.121764    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33274f28-f2d3-40fc-a16b-75e90ece007a-metrics-certs" pod "33274f28-f2d3-40fc-a16b-75e90ece007a" (UID: "33274f28-f2d3-40fc-a16b-75e90ece007a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:01:37.220901 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:37.220863    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33274f28-f2d3-40fc-a16b-75e90ece007a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33274f28-f2d3-40fc-a16b-75e90ece007a-maas-api-tls\") pod \"33274f28-f2d3-40fc-a16b-75e90ece007a\" (UID: \"33274f28-f2d3-40fc-a16b-75e90ece007a\") "
Nov 03 21:01:37.221334 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:37.221314    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33274f28-f2d3-40fc-a16b-75e90ece007a-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:37.221441 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:37.221338    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hnwv\" (UniqueName: \"kubernetes.io/projected/33274f28-f2d3-40fc-a16b-75e90ece007a-kube-api-access-5hnwv\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:37.223394 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:37.223365    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33274f28-f2d3-40fc-a16b-75e90ece007a-maas-api-tls" pod "33274f28-f2d3-40fc-a16b-75e90ece007a" (UID: "33274f28-f2d3-40fc-a16b-75e90ece007a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:01:37.322687 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:37.322578    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33274f28-f2d3-40fc-a16b-75e90ece007a-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:37.982107 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:37.982074    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq"
Nov 03 21:01:37.982107 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:37.982102    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq" event={"ID":"33274f28-f2d3-40fc-a16b-75e90ece007a","Type":"ContainerDied","Data":"39cde2908d2e15fd7d24343dbf88f208eb81f0787889403c04e5ce439c4564db"}
Nov 03 21:01:37.982660 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:37.982172    4421 scope.go:122] "RemoveContainer" containerID="56227fd565e459ced9ed09f25db9fd23e1542adbf202fd10c647f90c1566c64f"
Nov 03 21:01:38.005748 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:38.005713    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq"]
Nov 03 21:01:38.016379 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:38.016349    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-7a92d4-d78df6b4b-mtdsq"]
Nov 03 21:01:40.016012 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:40.015972    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33274f28-f2d3-40fc-a16b-75e90ece007a" path="/var/lib/kubelet/pods/33274f28-f2d3-40fc-a16b-75e90ece007a/volumes"
Nov 03 21:01:40.085630 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:40.082935    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"]
Nov 03 21:01:40.085630 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:40.083188    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb" podUID="4f55c349-f323-4dde-baab-4e279bfcf422" containerName="istio-proxy" containerID="cri-o://93e84ae9de24da2da7be2109e0b20ddbc51e6d050ad27cb119fe6dc1dc3a7aca" gracePeriod=30
Nov 03 21:01:45.766677 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.766652    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 21:01:45.882922 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.882842    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f55c349-f323-4dde-baab-4e279bfcf422-istio-token\" (UniqueName: \"kubernetes.io/projected/4f55c349-f323-4dde-baab-4e279bfcf422-istio-token\") pod \"4f55c349-f323-4dde-baab-4e279bfcf422\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") "
Nov 03 21:01:45.882922 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.882904    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f55c349-f323-4dde-baab-4e279bfcf422-kube-api-access-s5xtz\" (UniqueName: \"kubernetes.io/projected/4f55c349-f323-4dde-baab-4e279bfcf422-kube-api-access-s5xtz\") pod \"4f55c349-f323-4dde-baab-4e279bfcf422\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") "
Nov 03 21:01:45.883141 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.882944    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f55c349-f323-4dde-baab-4e279bfcf422-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f55c349-f323-4dde-baab-4e279bfcf422-istiod-ca-cert\") pod \"4f55c349-f323-4dde-baab-4e279bfcf422\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") "
Nov 03 21:01:45.883141 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.883006    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-workload-certs\") pod \"4f55c349-f323-4dde-baab-4e279bfcf422\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") "
Nov 03 21:01:45.883141 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.883099    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-istio-data\") pod \"4f55c349-f323-4dde-baab-4e279bfcf422\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") "
Nov 03 21:01:45.883332 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.883256    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-workload-socket\") pod \"4f55c349-f323-4dde-baab-4e279bfcf422\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") "
Nov 03 21:01:45.883460 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.883440    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f55c349-f323-4dde-baab-4e279bfcf422-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f55c349-f323-4dde-baab-4e279bfcf422-istio-podinfo\") pod \"4f55c349-f323-4dde-baab-4e279bfcf422\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") "
Nov 03 21:01:45.883527 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.883467    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-workload-certs" pod "4f55c349-f323-4dde-baab-4e279bfcf422" (UID: "4f55c349-f323-4dde-baab-4e279bfcf422"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:45.883778 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.883541    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-istio-envoy\") pod \"4f55c349-f323-4dde-baab-4e279bfcf422\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") "
Nov 03 21:01:45.883778 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.883621    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-credential-socket\") pod \"4f55c349-f323-4dde-baab-4e279bfcf422\" (UID: \"4f55c349-f323-4dde-baab-4e279bfcf422\") "
Nov 03 21:01:45.883778 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.883659    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f55c349-f323-4dde-baab-4e279bfcf422-istiod-ca-cert" pod "4f55c349-f323-4dde-baab-4e279bfcf422" (UID: "4f55c349-f323-4dde-baab-4e279bfcf422"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:01:45.883960 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.883805    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-istio-data" pod "4f55c349-f323-4dde-baab-4e279bfcf422" (UID: "4f55c349-f323-4dde-baab-4e279bfcf422"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:45.883960 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.883897    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-credential-socket" pod "4f55c349-f323-4dde-baab-4e279bfcf422" (UID: "4f55c349-f323-4dde-baab-4e279bfcf422"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:45.883960 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.883827    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-workload-socket" pod "4f55c349-f323-4dde-baab-4e279bfcf422" (UID: "4f55c349-f323-4dde-baab-4e279bfcf422"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:45.884523 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.884498    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:45.884665 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.884528    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f55c349-f323-4dde-baab-4e279bfcf422-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:45.884665 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.884545    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:45.884665 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.884559    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:45.884665 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.884573    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:45.885628 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.885570    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f55c349-f323-4dde-baab-4e279bfcf422-istio-token" pod "4f55c349-f323-4dde-baab-4e279bfcf422" (UID: "4f55c349-f323-4dde-baab-4e279bfcf422"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:01:45.886009 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.885987    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-istio-envoy" pod "4f55c349-f323-4dde-baab-4e279bfcf422" (UID: "4f55c349-f323-4dde-baab-4e279bfcf422"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:01:45.886318 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.886292    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f55c349-f323-4dde-baab-4e279bfcf422-kube-api-access-s5xtz" pod "4f55c349-f323-4dde-baab-4e279bfcf422" (UID: "4f55c349-f323-4dde-baab-4e279bfcf422"). InnerVolumeSpecName "kube-api-access-s5xtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:01:45.886445 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.886335    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f55c349-f323-4dde-baab-4e279bfcf422-istio-podinfo" pod "4f55c349-f323-4dde-baab-4e279bfcf422" (UID: "4f55c349-f323-4dde-baab-4e279bfcf422"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:01:45.986314 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.986279    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f55c349-f323-4dde-baab-4e279bfcf422-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:45.986314 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.986309    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f55c349-f323-4dde-baab-4e279bfcf422-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:45.986314 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.986320    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f55c349-f323-4dde-baab-4e279bfcf422-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:45.986576 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:45.986329    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5xtz\" (UniqueName: \"kubernetes.io/projected/4f55c349-f323-4dde-baab-4e279bfcf422-kube-api-access-s5xtz\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:46.040361 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:46.040330    4421 generic.go:356] "Generic (PLEG): container finished" podID="4f55c349-f323-4dde-baab-4e279bfcf422" containerID="93e84ae9de24da2da7be2109e0b20ddbc51e6d050ad27cb119fe6dc1dc3a7aca" exitCode=0
Nov 03 21:01:46.040517 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:46.040387    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"
Nov 03 21:01:46.040517 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:46.040403    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb" event={"ID":"4f55c349-f323-4dde-baab-4e279bfcf422","Type":"ContainerDied","Data":"93e84ae9de24da2da7be2109e0b20ddbc51e6d050ad27cb119fe6dc1dc3a7aca"}
Nov 03 21:01:46.040517 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:46.040477    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb" event={"ID":"4f55c349-f323-4dde-baab-4e279bfcf422","Type":"ContainerDied","Data":"4b3c3506f83ddd4b9cd5b07519515fcc77e47dd973994249bd9ac44a4501722c"}
Nov 03 21:01:46.040517 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:46.040501    4421 scope.go:122] "RemoveContainer" containerID="93e84ae9de24da2da7be2109e0b20ddbc51e6d050ad27cb119fe6dc1dc3a7aca"
Nov 03 21:01:46.053460 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:46.053438    4421 scope.go:122] "RemoveContainer" containerID="71a6959fc201244bd7cd969d7edf1cc4c88495e6e053bb2e90440313b33923cf"
Nov 03 21:01:46.063506 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:46.063485    4421 scope.go:122] "RemoveContainer" containerID="93e84ae9de24da2da7be2109e0b20ddbc51e6d050ad27cb119fe6dc1dc3a7aca"
Nov 03 21:01:46.063762 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:46.063742    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"]
Nov 03 21:01:46.064004 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:01:46.063984    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93e84ae9de24da2da7be2109e0b20ddbc51e6d050ad27cb119fe6dc1dc3a7aca\": container with ID starting with 93e84ae9de24da2da7be2109e0b20ddbc51e6d050ad27cb119fe6dc1dc3a7aca not found: ID does not exist" containerID="93e84ae9de24da2da7be2109e0b20ddbc51e6d050ad27cb119fe6dc1dc3a7aca"
Nov 03 21:01:46.064095 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:46.064017    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93e84ae9de24da2da7be2109e0b20ddbc51e6d050ad27cb119fe6dc1dc3a7aca"} err="failed to get container status \"93e84ae9de24da2da7be2109e0b20ddbc51e6d050ad27cb119fe6dc1dc3a7aca\": rpc error: code = NotFound desc = could not find container \"93e84ae9de24da2da7be2109e0b20ddbc51e6d050ad27cb119fe6dc1dc3a7aca\": container with ID starting with 93e84ae9de24da2da7be2109e0b20ddbc51e6d050ad27cb119fe6dc1dc3a7aca not found: ID does not exist"
Nov 03 21:01:46.064095 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:46.064042    4421 scope.go:122] "RemoveContainer" containerID="71a6959fc201244bd7cd969d7edf1cc4c88495e6e053bb2e90440313b33923cf"
Nov 03 21:01:46.064344 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:01:46.064328    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71a6959fc201244bd7cd969d7edf1cc4c88495e6e053bb2e90440313b33923cf\": container with ID starting with 71a6959fc201244bd7cd969d7edf1cc4c88495e6e053bb2e90440313b33923cf not found: ID does not exist" containerID="71a6959fc201244bd7cd969d7edf1cc4c88495e6e053bb2e90440313b33923cf"
Nov 03 21:01:46.064382 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:46.064350    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71a6959fc201244bd7cd969d7edf1cc4c88495e6e053bb2e90440313b33923cf"} err="failed to get container status \"71a6959fc201244bd7cd969d7edf1cc4c88495e6e053bb2e90440313b33923cf\": rpc error: code = NotFound desc = could not find container \"71a6959fc201244bd7cd969d7edf1cc4c88495e6e053bb2e90440313b33923cf\": container with ID starting with 71a6959fc201244bd7cd969d7edf1cc4c88495e6e053bb2e90440313b33923cf not found: ID does not exist"
Nov 03 21:01:46.069230 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:46.069211    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7a92d4-openshift-default-6cff4dfcbc-splnb"]
Nov 03 21:01:46.277001 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:46.276959    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz"]
Nov 03 21:01:46.325715 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:46.325684    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj"]
Nov 03 21:01:48.016970 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:48.016938    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f55c349-f323-4dde-baab-4e279bfcf422" path="/var/lib/kubelet/pods/4f55c349-f323-4dde-baab-4e279bfcf422/volumes"
Nov 03 21:01:49.628936 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:01:49.628846    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:01:51.278578 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:51.278504    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz" podUID="2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8" containerName="payload-processing" containerID="cri-o://9234a64384bc59f6b6ea26851255c46133d9ed9612573ab109e7f849caf61af4" gracePeriod=25
Nov 03 21:01:51.328685 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:51.328642    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj" podUID="3b3e4444-3d44-4623-94a7-d4748d93a6e7" containerName="payload-pre-processing" containerID="cri-o://739f25c7d5f02607fa6c77b3829634de38ae9c9c2db3030db17af97718a7ddf8" gracePeriod=25
Nov 03 21:01:51.627071 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:51.627045    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz"
Nov 03 21:01:51.685493 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:51.685453    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8-plugins-config-volume\") pod \"2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8\" (UID: \"2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8\") "
Nov 03 21:01:51.685915 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:51.685512    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8-kube-api-access-nqglr\" (UniqueName: \"kubernetes.io/projected/2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8-kube-api-access-nqglr\") pod \"2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8\" (UID: \"2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8\") "
Nov 03 21:01:51.686450 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:51.686420    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8-plugins-config-volume" pod "2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8" (UID: "2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:01:51.687057 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:51.687032    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:51.689080 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:51.689053    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8-kube-api-access-nqglr" pod "2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8" (UID: "2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8"). InnerVolumeSpecName "kube-api-access-nqglr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:01:51.698677 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:51.698657    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj"
Nov 03 21:01:51.788510 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:51.788475    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b3e4444-3d44-4623-94a7-d4748d93a6e7-kube-api-access-6gnfs\" (UniqueName: \"kubernetes.io/projected/3b3e4444-3d44-4623-94a7-d4748d93a6e7-kube-api-access-6gnfs\") pod \"3b3e4444-3d44-4623-94a7-d4748d93a6e7\" (UID: \"3b3e4444-3d44-4623-94a7-d4748d93a6e7\") "
Nov 03 21:01:51.788799 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:51.788772    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b3e4444-3d44-4623-94a7-d4748d93a6e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b3e4444-3d44-4623-94a7-d4748d93a6e7-plugins-config-volume\") pod \"3b3e4444-3d44-4623-94a7-d4748d93a6e7\" (UID: \"3b3e4444-3d44-4623-94a7-d4748d93a6e7\") "
Nov 03 21:01:51.789194 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:51.789166    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b3e4444-3d44-4623-94a7-d4748d93a6e7-plugins-config-volume" pod "3b3e4444-3d44-4623-94a7-d4748d93a6e7" (UID: "3b3e4444-3d44-4623-94a7-d4748d93a6e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:01:51.789456 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:51.789299    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b3e4444-3d44-4623-94a7-d4748d93a6e7-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:51.789456 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:51.789313    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqglr\" (UniqueName: \"kubernetes.io/projected/2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8-kube-api-access-nqglr\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:51.790940 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:51.790917    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b3e4444-3d44-4623-94a7-d4748d93a6e7-kube-api-access-6gnfs" pod "3b3e4444-3d44-4623-94a7-d4748d93a6e7" (UID: "3b3e4444-3d44-4623-94a7-d4748d93a6e7"). InnerVolumeSpecName "kube-api-access-6gnfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:01:51.891408 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:51.891336    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gnfs\" (UniqueName: \"kubernetes.io/projected/3b3e4444-3d44-4623-94a7-d4748d93a6e7-kube-api-access-6gnfs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:52.081789 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:52.081755    4421 generic.go:356] "Generic (PLEG): container finished" podID="3b3e4444-3d44-4623-94a7-d4748d93a6e7" containerID="739f25c7d5f02607fa6c77b3829634de38ae9c9c2db3030db17af97718a7ddf8" exitCode=0
Nov 03 21:01:52.082004 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:52.081805    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj"
Nov 03 21:01:52.082004 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:52.081844    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj" event={"ID":"3b3e4444-3d44-4623-94a7-d4748d93a6e7","Type":"ContainerDied","Data":"739f25c7d5f02607fa6c77b3829634de38ae9c9c2db3030db17af97718a7ddf8"}
Nov 03 21:01:52.082004 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:52.081897    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj" event={"ID":"3b3e4444-3d44-4623-94a7-d4748d93a6e7","Type":"ContainerDied","Data":"b319357306b812fca8cf738e9c2fff5b5dc81b056b42aff3741fdf0ea2399d38"}
Nov 03 21:01:52.082004 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:52.081925    4421 scope.go:122] "RemoveContainer" containerID="739f25c7d5f02607fa6c77b3829634de38ae9c9c2db3030db17af97718a7ddf8"
Nov 03 21:01:52.083392 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:52.083368    4421 generic.go:356] "Generic (PLEG): container finished" podID="2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8" containerID="9234a64384bc59f6b6ea26851255c46133d9ed9612573ab109e7f849caf61af4" exitCode=0
Nov 03 21:01:52.083516 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:52.083407    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz" event={"ID":"2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8","Type":"ContainerDied","Data":"9234a64384bc59f6b6ea26851255c46133d9ed9612573ab109e7f849caf61af4"}
Nov 03 21:01:52.083516 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:52.083431    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz"
Nov 03 21:01:52.083516 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:52.083448    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz" event={"ID":"2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8","Type":"ContainerDied","Data":"e48ee52fbd406fc429aa26d7203c9c692e0359d6fabb5699ea98ed46fbcbf2ec"}
Nov 03 21:01:52.092737 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:52.092719    4421 scope.go:122] "RemoveContainer" containerID="739f25c7d5f02607fa6c77b3829634de38ae9c9c2db3030db17af97718a7ddf8"
Nov 03 21:01:52.093119 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:01:52.093098    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"739f25c7d5f02607fa6c77b3829634de38ae9c9c2db3030db17af97718a7ddf8\": container with ID starting with 739f25c7d5f02607fa6c77b3829634de38ae9c9c2db3030db17af97718a7ddf8 not found: ID does not exist" containerID="739f25c7d5f02607fa6c77b3829634de38ae9c9c2db3030db17af97718a7ddf8"
Nov 03 21:01:52.093219 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:52.093126    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"739f25c7d5f02607fa6c77b3829634de38ae9c9c2db3030db17af97718a7ddf8"} err="failed to get container status \"739f25c7d5f02607fa6c77b3829634de38ae9c9c2db3030db17af97718a7ddf8\": rpc error: code = NotFound desc = could not find container \"739f25c7d5f02607fa6c77b3829634de38ae9c9c2db3030db17af97718a7ddf8\": container with ID starting with 739f25c7d5f02607fa6c77b3829634de38ae9c9c2db3030db17af97718a7ddf8 not found: ID does not exist"
Nov 03 21:01:52.093219 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:52.093142    4421 scope.go:122] "RemoveContainer" containerID="9234a64384bc59f6b6ea26851255c46133d9ed9612573ab109e7f849caf61af4"
Nov 03 21:01:52.103811 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:52.103792    4421 scope.go:122] "RemoveContainer" containerID="9234a64384bc59f6b6ea26851255c46133d9ed9612573ab109e7f849caf61af4"
Nov 03 21:01:52.104101 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:01:52.104082    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9234a64384bc59f6b6ea26851255c46133d9ed9612573ab109e7f849caf61af4\": container with ID starting with 9234a64384bc59f6b6ea26851255c46133d9ed9612573ab109e7f849caf61af4 not found: ID does not exist" containerID="9234a64384bc59f6b6ea26851255c46133d9ed9612573ab109e7f849caf61af4"
Nov 03 21:01:52.104181 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:52.104107    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9234a64384bc59f6b6ea26851255c46133d9ed9612573ab109e7f849caf61af4"} err="failed to get container status \"9234a64384bc59f6b6ea26851255c46133d9ed9612573ab109e7f849caf61af4\": rpc error: code = NotFound desc = could not find container \"9234a64384bc59f6b6ea26851255c46133d9ed9612573ab109e7f849caf61af4\": container with ID starting with 9234a64384bc59f6b6ea26851255c46133d9ed9612573ab109e7f849caf61af4 not found: ID does not exist"
Nov 03 21:01:52.106938 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:52.106913    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj"]
Nov 03 21:01:52.109816 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:52.109787    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-08a7bc-76646b7d95-hvfxj"]
Nov 03 21:01:52.120866 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:52.120843    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz"]
Nov 03 21:01:52.127182 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:52.127164    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-08a7bc-859d5b9d86-s56vz"]
Nov 03 21:01:54.016040 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:54.016000    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8" path="/var/lib/kubelet/pods/2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8/volumes"
Nov 03 21:01:54.016515 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:54.016443    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b3e4444-3d44-4623-94a7-d4748d93a6e7" path="/var/lib/kubelet/pods/3b3e4444-3d44-4623-94a7-d4748d93a6e7/volumes"
Nov 03 21:01:57.041653 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:57.041618    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q"]
Nov 03 21:01:57.042430 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:57.042379    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q" podUID="38f64ec2-df7a-4c4e-bdd5-d25f13125ad6" containerName="maas-api" containerID="cri-o://29fcbe7958e6c047827c58676aa8a4b9d6747d50fc67531a0b6cddc61e52979b" gracePeriod=30
Nov 03 21:01:57.413781 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:57.413752    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q"
Nov 03 21:01:57.487321 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:57.487279    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-maas-api-tls\") pod \"38f64ec2-df7a-4c4e-bdd5-d25f13125ad6\" (UID: \"38f64ec2-df7a-4c4e-bdd5-d25f13125ad6\") "
Nov 03 21:01:57.487540 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:57.487395    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-kube-api-access-g46zn\" (UniqueName: \"kubernetes.io/projected/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-kube-api-access-g46zn\") pod \"38f64ec2-df7a-4c4e-bdd5-d25f13125ad6\" (UID: \"38f64ec2-df7a-4c4e-bdd5-d25f13125ad6\") "
Nov 03 21:01:57.487540 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:57.487447    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-openshift-service-ca\") pod \"38f64ec2-df7a-4c4e-bdd5-d25f13125ad6\" (UID: \"38f64ec2-df7a-4c4e-bdd5-d25f13125ad6\") "
Nov 03 21:01:57.487540 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:57.487498    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-metrics-certs\") pod \"38f64ec2-df7a-4c4e-bdd5-d25f13125ad6\" (UID: \"38f64ec2-df7a-4c4e-bdd5-d25f13125ad6\") "
Nov 03 21:01:57.489073 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:57.489044    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-openshift-service-ca" pod "38f64ec2-df7a-4c4e-bdd5-d25f13125ad6" (UID: "38f64ec2-df7a-4c4e-bdd5-d25f13125ad6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:01:57.492836 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:57.492793    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-metrics-certs" pod "38f64ec2-df7a-4c4e-bdd5-d25f13125ad6" (UID: "38f64ec2-df7a-4c4e-bdd5-d25f13125ad6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:01:57.492940 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:57.492832    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-kube-api-access-g46zn" pod "38f64ec2-df7a-4c4e-bdd5-d25f13125ad6" (UID: "38f64ec2-df7a-4c4e-bdd5-d25f13125ad6"). InnerVolumeSpecName "kube-api-access-g46zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:01:57.497317 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:57.497286    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-maas-api-tls" pod "38f64ec2-df7a-4c4e-bdd5-d25f13125ad6" (UID: "38f64ec2-df7a-4c4e-bdd5-d25f13125ad6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:01:57.589077 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:57.589041    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:57.589077 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:57.589071    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:57.589077 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:57.589080    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:57.589355 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:57.589089    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g46zn\" (UniqueName: \"kubernetes.io/projected/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6-kube-api-access-g46zn\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:01:58.125535 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:58.125498    4421 generic.go:356] "Generic (PLEG): container finished" podID="38f64ec2-df7a-4c4e-bdd5-d25f13125ad6" containerID="29fcbe7958e6c047827c58676aa8a4b9d6747d50fc67531a0b6cddc61e52979b" exitCode=0
Nov 03 21:01:58.126108 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:58.125691    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q" event={"ID":"38f64ec2-df7a-4c4e-bdd5-d25f13125ad6","Type":"ContainerDied","Data":"29fcbe7958e6c047827c58676aa8a4b9d6747d50fc67531a0b6cddc61e52979b"}
Nov 03 21:01:58.126108 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:58.125746    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q" event={"ID":"38f64ec2-df7a-4c4e-bdd5-d25f13125ad6","Type":"ContainerDied","Data":"2c973ce6da576727ea988c0f4238a236de3a3ab55f66ae27b091119a48c60f9b"}
Nov 03 21:01:58.126108 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:58.125777    4421 scope.go:122] "RemoveContainer" containerID="29fcbe7958e6c047827c58676aa8a4b9d6747d50fc67531a0b6cddc61e52979b"
Nov 03 21:01:58.126108 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:58.125933    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q"
Nov 03 21:01:58.161128 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:58.161082    4421 scope.go:122] "RemoveContainer" containerID="29fcbe7958e6c047827c58676aa8a4b9d6747d50fc67531a0b6cddc61e52979b"
Nov 03 21:01:58.161289 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:58.161210    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q"]
Nov 03 21:01:58.161576 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:01:58.161543    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29fcbe7958e6c047827c58676aa8a4b9d6747d50fc67531a0b6cddc61e52979b\": container with ID starting with 29fcbe7958e6c047827c58676aa8a4b9d6747d50fc67531a0b6cddc61e52979b not found: ID does not exist" containerID="29fcbe7958e6c047827c58676aa8a4b9d6747d50fc67531a0b6cddc61e52979b"
Nov 03 21:01:58.161712 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:58.161586    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29fcbe7958e6c047827c58676aa8a4b9d6747d50fc67531a0b6cddc61e52979b"} err="failed to get container status \"29fcbe7958e6c047827c58676aa8a4b9d6747d50fc67531a0b6cddc61e52979b\": rpc error: code = NotFound desc = could not find container \"29fcbe7958e6c047827c58676aa8a4b9d6747d50fc67531a0b6cddc61e52979b\": container with ID starting with 29fcbe7958e6c047827c58676aa8a4b9d6747d50fc67531a0b6cddc61e52979b not found: ID does not exist"
Nov 03 21:01:58.164725 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:01:58.164699    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-08a7bc-7d5455fbff-kh98q"]
Nov 03 21:02:00.016981 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:00.016946    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38f64ec2-df7a-4c4e-bdd5-d25f13125ad6" path="/var/lib/kubelet/pods/38f64ec2-df7a-4c4e-bdd5-d25f13125ad6/volumes"
Nov 03 21:02:07.165471 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:07.165435    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"]
Nov 03 21:02:12.188316 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:12.188278    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw"]
Nov 03 21:02:12.211813 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:12.211780    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq"]
Nov 03 21:02:13.511198 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:13.511162    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww"]
Nov 03 21:02:17.188756 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:17.188712    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw" podUID="0ae207be-6a8f-4773-bc31-527edb88be5e" containerName="payload-processing" containerID="cri-o://290d215e8ff95668de33fe2bf943b9b7bb56fcf991aaaae62de907aa9d462f59" gracePeriod=25
Nov 03 21:02:17.222431 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:17.222395    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq" podUID="3a0dff8b-05ed-4261-a30c-08ced3980a2c" containerName="payload-pre-processing" containerID="cri-o://ea0e38b5dc46921faeac45d30eeee38c55748ba3462c9a77155a331e8c5d2b80" gracePeriod=25
Nov 03 21:02:17.470029 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:17.470005    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw"
Nov 03 21:02:17.499764 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:17.499734    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq" podUID="3a0dff8b-05ed-4261-a30c-08ced3980a2c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:02:17.507133 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:17.507101    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ae207be-6a8f-4773-bc31-527edb88be5e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae207be-6a8f-4773-bc31-527edb88be5e-plugins-config-volume\") pod \"0ae207be-6a8f-4773-bc31-527edb88be5e\" (UID: \"0ae207be-6a8f-4773-bc31-527edb88be5e\") "
Nov 03 21:02:17.507308 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:17.507213    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ae207be-6a8f-4773-bc31-527edb88be5e-kube-api-access-c5hqg\" (UniqueName: \"kubernetes.io/projected/0ae207be-6a8f-4773-bc31-527edb88be5e-kube-api-access-c5hqg\") pod \"0ae207be-6a8f-4773-bc31-527edb88be5e\" (UID: \"0ae207be-6a8f-4773-bc31-527edb88be5e\") "
Nov 03 21:02:17.508333 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:17.508287    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ae207be-6a8f-4773-bc31-527edb88be5e-plugins-config-volume" pod "0ae207be-6a8f-4773-bc31-527edb88be5e" (UID: "0ae207be-6a8f-4773-bc31-527edb88be5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:02:17.510207 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:17.510184    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae207be-6a8f-4773-bc31-527edb88be5e-kube-api-access-c5hqg" pod "0ae207be-6a8f-4773-bc31-527edb88be5e" (UID: "0ae207be-6a8f-4773-bc31-527edb88be5e"). InnerVolumeSpecName "kube-api-access-c5hqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:02:17.608930 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:17.608897    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae207be-6a8f-4773-bc31-527edb88be5e-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:17.608930 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:17.608925    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5hqg\" (UniqueName: \"kubernetes.io/projected/0ae207be-6a8f-4773-bc31-527edb88be5e-kube-api-access-c5hqg\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:18.275704 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:18.275666    4421 generic.go:356] "Generic (PLEG): container finished" podID="0ae207be-6a8f-4773-bc31-527edb88be5e" containerID="290d215e8ff95668de33fe2bf943b9b7bb56fcf991aaaae62de907aa9d462f59" exitCode=0
Nov 03 21:02:18.276173 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:18.275715    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw"
Nov 03 21:02:18.276173 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:18.275753    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw" event={"ID":"0ae207be-6a8f-4773-bc31-527edb88be5e","Type":"ContainerDied","Data":"290d215e8ff95668de33fe2bf943b9b7bb56fcf991aaaae62de907aa9d462f59"}
Nov 03 21:02:18.276173 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:18.275809    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw" event={"ID":"0ae207be-6a8f-4773-bc31-527edb88be5e","Type":"ContainerDied","Data":"f3e88cec57f935da13825fe505c78be7203932dfd221a48ad73448824d5cfd41"}
Nov 03 21:02:18.276173 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:18.275830    4421 scope.go:122] "RemoveContainer" containerID="290d215e8ff95668de33fe2bf943b9b7bb56fcf991aaaae62de907aa9d462f59"
Nov 03 21:02:18.287742 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:18.287714    4421 scope.go:122] "RemoveContainer" containerID="290d215e8ff95668de33fe2bf943b9b7bb56fcf991aaaae62de907aa9d462f59"
Nov 03 21:02:18.288083 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:02:18.288062    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"290d215e8ff95668de33fe2bf943b9b7bb56fcf991aaaae62de907aa9d462f59\": container with ID starting with 290d215e8ff95668de33fe2bf943b9b7bb56fcf991aaaae62de907aa9d462f59 not found: ID does not exist" containerID="290d215e8ff95668de33fe2bf943b9b7bb56fcf991aaaae62de907aa9d462f59"
Nov 03 21:02:18.288156 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:18.288091    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"290d215e8ff95668de33fe2bf943b9b7bb56fcf991aaaae62de907aa9d462f59"} err="failed to get container status \"290d215e8ff95668de33fe2bf943b9b7bb56fcf991aaaae62de907aa9d462f59\": rpc error: code = NotFound desc = could not find container \"290d215e8ff95668de33fe2bf943b9b7bb56fcf991aaaae62de907aa9d462f59\": container with ID starting with 290d215e8ff95668de33fe2bf943b9b7bb56fcf991aaaae62de907aa9d462f59 not found: ID does not exist"
Nov 03 21:02:18.296615 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:18.296575    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw"]
Nov 03 21:02:18.300977 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:18.300949    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-14ebbb-5fb74ff66d-s9rdw"]
Nov 03 21:02:18.513237 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:18.513024    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww" podUID="5107af0d-00d7-415e-be8e-e9ce4d55f8c0" containerName="payload-pre-processing" containerID="cri-o://d97ee714e89844c09b2c11978f09d50562d9c4e5b58a809deb0a41176784122d" gracePeriod=25
Nov 03 21:02:18.794958 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:18.794916    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww"
Nov 03 21:02:18.828970 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:18.828941    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5107af0d-00d7-415e-be8e-e9ce4d55f8c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5107af0d-00d7-415e-be8e-e9ce4d55f8c0-plugins-config-volume\") pod \"5107af0d-00d7-415e-be8e-e9ce4d55f8c0\" (UID: \"5107af0d-00d7-415e-be8e-e9ce4d55f8c0\") "
Nov 03 21:02:18.829155 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:18.829091    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5107af0d-00d7-415e-be8e-e9ce4d55f8c0-kube-api-access-kzrd8\" (UniqueName: \"kubernetes.io/projected/5107af0d-00d7-415e-be8e-e9ce4d55f8c0-kube-api-access-kzrd8\") pod \"5107af0d-00d7-415e-be8e-e9ce4d55f8c0\" (UID: \"5107af0d-00d7-415e-be8e-e9ce4d55f8c0\") "
Nov 03 21:02:18.829357 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:18.829334    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5107af0d-00d7-415e-be8e-e9ce4d55f8c0-plugins-config-volume" pod "5107af0d-00d7-415e-be8e-e9ce4d55f8c0" (UID: "5107af0d-00d7-415e-be8e-e9ce4d55f8c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:02:18.829670 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:18.829650    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5107af0d-00d7-415e-be8e-e9ce4d55f8c0-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:18.831367 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:18.831336    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5107af0d-00d7-415e-be8e-e9ce4d55f8c0-kube-api-access-kzrd8" pod "5107af0d-00d7-415e-be8e-e9ce4d55f8c0" (UID: "5107af0d-00d7-415e-be8e-e9ce4d55f8c0"). InnerVolumeSpecName "kube-api-access-kzrd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:02:18.931494 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:18.931422    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzrd8\" (UniqueName: \"kubernetes.io/projected/5107af0d-00d7-415e-be8e-e9ce4d55f8c0-kube-api-access-kzrd8\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:19.285476 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:19.285438    4421 generic.go:356] "Generic (PLEG): container finished" podID="5107af0d-00d7-415e-be8e-e9ce4d55f8c0" containerID="d97ee714e89844c09b2c11978f09d50562d9c4e5b58a809deb0a41176784122d" exitCode=0
Nov 03 21:02:19.285999 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:19.285573    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww" event={"ID":"5107af0d-00d7-415e-be8e-e9ce4d55f8c0","Type":"ContainerDied","Data":"d97ee714e89844c09b2c11978f09d50562d9c4e5b58a809deb0a41176784122d"}
Nov 03 21:02:19.285999 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:19.285648    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww" event={"ID":"5107af0d-00d7-415e-be8e-e9ce4d55f8c0","Type":"ContainerDied","Data":"52be822881c58ec844b7e189f8f8dfb53de1d4c889c967650929ee3bcc3c2bf3"}
Nov 03 21:02:19.285999 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:19.285670    4421 scope.go:122] "RemoveContainer" containerID="d97ee714e89844c09b2c11978f09d50562d9c4e5b58a809deb0a41176784122d"
Nov 03 21:02:19.285999 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:19.285772    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww"
Nov 03 21:02:19.303799 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:19.303776    4421 scope.go:122] "RemoveContainer" containerID="d97ee714e89844c09b2c11978f09d50562d9c4e5b58a809deb0a41176784122d"
Nov 03 21:02:19.304075 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:02:19.304054    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d97ee714e89844c09b2c11978f09d50562d9c4e5b58a809deb0a41176784122d\": container with ID starting with d97ee714e89844c09b2c11978f09d50562d9c4e5b58a809deb0a41176784122d not found: ID does not exist" containerID="d97ee714e89844c09b2c11978f09d50562d9c4e5b58a809deb0a41176784122d"
Nov 03 21:02:19.304144 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:19.304085    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97ee714e89844c09b2c11978f09d50562d9c4e5b58a809deb0a41176784122d"} err="failed to get container status \"d97ee714e89844c09b2c11978f09d50562d9c4e5b58a809deb0a41176784122d\": rpc error: code = NotFound desc = could not find container \"d97ee714e89844c09b2c11978f09d50562d9c4e5b58a809deb0a41176784122d\": container with ID starting with d97ee714e89844c09b2c11978f09d50562d9c4e5b58a809deb0a41176784122d not found: ID does not exist"
Nov 03 21:02:19.319247 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:19.319212    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww"]
Nov 03 21:02:19.326394 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:19.326366    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-b13a14-7df99d5455-qf9ww"]
Nov 03 21:02:19.631284 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:02:19.631191    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:02:20.017661 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:20.017628    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ae207be-6a8f-4773-bc31-527edb88be5e" path="/var/lib/kubelet/pods/0ae207be-6a8f-4773-bc31-527edb88be5e/volumes"
Nov 03 21:02:20.018103 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:20.018089    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5107af0d-00d7-415e-be8e-e9ce4d55f8c0" path="/var/lib/kubelet/pods/5107af0d-00d7-415e-be8e-e9ce4d55f8c0/volumes"
Nov 03 21:02:22.204312 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:22.204264    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main" containerID="cri-o://af60b8e04fc5efed6264f40614db441743fa2556a13e5562846220f344eebd0a" gracePeriod=45
Nov 03 21:02:22.556151 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:22.556123    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 21:02:22.594707 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:22.594483    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7be6b1b-1739-448a-9ca7-27556b447c0f-kube-api-access-hwxcg\" (UniqueName: \"kubernetes.io/projected/b7be6b1b-1739-448a-9ca7-27556b447c0f-kube-api-access-hwxcg\") pod \"b7be6b1b-1739-448a-9ca7-27556b447c0f\" (UID: \"b7be6b1b-1739-448a-9ca7-27556b447c0f\") "
Nov 03 21:02:22.594707 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:22.594541    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-model-cache\") pod \"b7be6b1b-1739-448a-9ca7-27556b447c0f\" (UID: \"b7be6b1b-1739-448a-9ca7-27556b447c0f\") "
Nov 03 21:02:22.594707 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:22.594616    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-dshm\") pod \"b7be6b1b-1739-448a-9ca7-27556b447c0f\" (UID: \"b7be6b1b-1739-448a-9ca7-27556b447c0f\") "
Nov 03 21:02:22.595197 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:22.595165    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-tmp-dir\") pod \"b7be6b1b-1739-448a-9ca7-27556b447c0f\" (UID: \"b7be6b1b-1739-448a-9ca7-27556b447c0f\") "
Nov 03 21:02:22.595412 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:22.595383    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7be6b1b-1739-448a-9ca7-27556b447c0f-tls-certs\" (UniqueName: \"kubernetes.io/secret/b7be6b1b-1739-448a-9ca7-27556b447c0f-tls-certs\") pod \"b7be6b1b-1739-448a-9ca7-27556b447c0f\" (UID: \"b7be6b1b-1739-448a-9ca7-27556b447c0f\") "
Nov 03 21:02:22.595893 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:22.595538    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-home\" (UniqueName: \"kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-home\") pod \"b7be6b1b-1739-448a-9ca7-27556b447c0f\" (UID: \"b7be6b1b-1739-448a-9ca7-27556b447c0f\") "
Nov 03 21:02:22.596542 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:22.596516    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-home" pod "b7be6b1b-1739-448a-9ca7-27556b447c0f" (UID: "b7be6b1b-1739-448a-9ca7-27556b447c0f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:02:22.599490 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:22.599193    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-model-cache" pod "b7be6b1b-1739-448a-9ca7-27556b447c0f" (UID: "b7be6b1b-1739-448a-9ca7-27556b447c0f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:02:22.599490 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:22.599471    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-tmp-dir" pod "b7be6b1b-1739-448a-9ca7-27556b447c0f" (UID: "b7be6b1b-1739-448a-9ca7-27556b447c0f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:02:22.602650 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:22.600202    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7be6b1b-1739-448a-9ca7-27556b447c0f-kube-api-access-hwxcg" pod "b7be6b1b-1739-448a-9ca7-27556b447c0f" (UID: "b7be6b1b-1739-448a-9ca7-27556b447c0f"). InnerVolumeSpecName "kube-api-access-hwxcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:02:22.602650 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:22.602037    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-dshm" pod "b7be6b1b-1739-448a-9ca7-27556b447c0f" (UID: "b7be6b1b-1739-448a-9ca7-27556b447c0f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:02:22.602650 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:22.602155    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7be6b1b-1739-448a-9ca7-27556b447c0f-tls-certs" pod "b7be6b1b-1739-448a-9ca7-27556b447c0f" (UID: "b7be6b1b-1739-448a-9ca7-27556b447c0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:02:22.698000 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:22.697965    4421 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7be6b1b-1739-448a-9ca7-27556b447c0f-tls-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:22.698195 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:22.698008    4421 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-home\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:22.698195 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:22.698025    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwxcg\" (UniqueName: \"kubernetes.io/projected/b7be6b1b-1739-448a-9ca7-27556b447c0f-kube-api-access-hwxcg\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:22.698195 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:22.698038    4421 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-model-cache\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:22.698195 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:22.698050    4421 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-dshm\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:22.698195 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:22.698061    4421 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7be6b1b-1739-448a-9ca7-27556b447c0f-tmp-dir\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:23.372066 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:23.372032    4421 generic.go:356] "Generic (PLEG): container finished" podID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerID="af60b8e04fc5efed6264f40614db441743fa2556a13e5562846220f344eebd0a" exitCode=0
Nov 03 21:02:23.372515 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:23.372080    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"
Nov 03 21:02:23.372515 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:23.372096    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" event={"ID":"b7be6b1b-1739-448a-9ca7-27556b447c0f","Type":"ContainerDied","Data":"af60b8e04fc5efed6264f40614db441743fa2556a13e5562846220f344eebd0a"}
Nov 03 21:02:23.372515 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:23.372133    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r" event={"ID":"b7be6b1b-1739-448a-9ca7-27556b447c0f","Type":"ContainerDied","Data":"69ec73d098771c1c182352c54047a009d1125141772c2c865eed31291cce8d03"}
Nov 03 21:02:23.372515 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:23.372156    4421 scope.go:122] "RemoveContainer" containerID="af60b8e04fc5efed6264f40614db441743fa2556a13e5562846220f344eebd0a"
Nov 03 21:02:23.383320 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:23.383297    4421 scope.go:122] "RemoveContainer" containerID="af60b8e04fc5efed6264f40614db441743fa2556a13e5562846220f344eebd0a"
Nov 03 21:02:23.383613 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:02:23.383578    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af60b8e04fc5efed6264f40614db441743fa2556a13e5562846220f344eebd0a\": container with ID starting with af60b8e04fc5efed6264f40614db441743fa2556a13e5562846220f344eebd0a not found: ID does not exist" containerID="af60b8e04fc5efed6264f40614db441743fa2556a13e5562846220f344eebd0a"
Nov 03 21:02:23.383684 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:23.383638    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af60b8e04fc5efed6264f40614db441743fa2556a13e5562846220f344eebd0a"} err="failed to get container status \"af60b8e04fc5efed6264f40614db441743fa2556a13e5562846220f344eebd0a\": rpc error: code = NotFound desc = could not find container \"af60b8e04fc5efed6264f40614db441743fa2556a13e5562846220f344eebd0a\": container with ID starting with af60b8e04fc5efed6264f40614db441743fa2556a13e5562846220f344eebd0a not found: ID does not exist"
Nov 03 21:02:23.396556 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:23.396526    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"]
Nov 03 21:02:23.402879 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:23.402855    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b13a14/distinct-2-w1-b13a14-kserve-5c7648fb99-pf76r"]
Nov 03 21:02:24.016491 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:24.016449    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" path="/var/lib/kubelet/pods/b7be6b1b-1739-448a-9ca7-27556b447c0f/volumes"
Nov 03 21:02:27.499847 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:27.499803    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq" podUID="3a0dff8b-05ed-4261-a30c-08ced3980a2c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:02:31.502022 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:31.501982    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4"]
Nov 03 21:02:31.502555 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:31.502247    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4" podUID="070c9447-94dc-4af0-90a1-ad328f85bd84" containerName="maas-api" containerID="cri-o://3c9c7cb8c51c1f3f643938ca775e6cfff7b104b236027c262c5ba658cad15bcc" gracePeriod=30
Nov 03 21:02:31.896694 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:31.896669    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4"
Nov 03 21:02:31.968259 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:31.968229    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/070c9447-94dc-4af0-90a1-ad328f85bd84-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/070c9447-94dc-4af0-90a1-ad328f85bd84-openshift-service-ca\") pod \"070c9447-94dc-4af0-90a1-ad328f85bd84\" (UID: \"070c9447-94dc-4af0-90a1-ad328f85bd84\") "
Nov 03 21:02:31.968476 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:31.968300    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/070c9447-94dc-4af0-90a1-ad328f85bd84-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/070c9447-94dc-4af0-90a1-ad328f85bd84-maas-api-tls\") pod \"070c9447-94dc-4af0-90a1-ad328f85bd84\" (UID: \"070c9447-94dc-4af0-90a1-ad328f85bd84\") "
Nov 03 21:02:31.968476 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:31.968367    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/070c9447-94dc-4af0-90a1-ad328f85bd84-metrics-certs\" (UniqueName: \"kubernetes.io/secret/070c9447-94dc-4af0-90a1-ad328f85bd84-metrics-certs\") pod \"070c9447-94dc-4af0-90a1-ad328f85bd84\" (UID: \"070c9447-94dc-4af0-90a1-ad328f85bd84\") "
Nov 03 21:02:31.968637 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:31.968482    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/070c9447-94dc-4af0-90a1-ad328f85bd84-kube-api-access-mpwqb\" (UniqueName: \"kubernetes.io/projected/070c9447-94dc-4af0-90a1-ad328f85bd84-kube-api-access-mpwqb\") pod \"070c9447-94dc-4af0-90a1-ad328f85bd84\" (UID: \"070c9447-94dc-4af0-90a1-ad328f85bd84\") "
Nov 03 21:02:31.968705 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:31.968647    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/070c9447-94dc-4af0-90a1-ad328f85bd84-openshift-service-ca" pod "070c9447-94dc-4af0-90a1-ad328f85bd84" (UID: "070c9447-94dc-4af0-90a1-ad328f85bd84"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:02:31.969122 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:31.969100    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/070c9447-94dc-4af0-90a1-ad328f85bd84-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:31.970751 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:31.970726    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070c9447-94dc-4af0-90a1-ad328f85bd84-kube-api-access-mpwqb" pod "070c9447-94dc-4af0-90a1-ad328f85bd84" (UID: "070c9447-94dc-4af0-90a1-ad328f85bd84"). InnerVolumeSpecName "kube-api-access-mpwqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:02:31.970863 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:31.970773    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/070c9447-94dc-4af0-90a1-ad328f85bd84-metrics-certs" pod "070c9447-94dc-4af0-90a1-ad328f85bd84" (UID: "070c9447-94dc-4af0-90a1-ad328f85bd84"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:02:31.970863 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:31.970789    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/070c9447-94dc-4af0-90a1-ad328f85bd84-maas-api-tls" pod "070c9447-94dc-4af0-90a1-ad328f85bd84" (UID: "070c9447-94dc-4af0-90a1-ad328f85bd84"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:02:32.070571 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:32.070539    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/070c9447-94dc-4af0-90a1-ad328f85bd84-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:32.070571 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:32.070563    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/070c9447-94dc-4af0-90a1-ad328f85bd84-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:32.070571 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:32.070574    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpwqb\" (UniqueName: \"kubernetes.io/projected/070c9447-94dc-4af0-90a1-ad328f85bd84-kube-api-access-mpwqb\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:32.431809 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:32.431777    4421 generic.go:356] "Generic (PLEG): container finished" podID="070c9447-94dc-4af0-90a1-ad328f85bd84" containerID="3c9c7cb8c51c1f3f643938ca775e6cfff7b104b236027c262c5ba658cad15bcc" exitCode=0
Nov 03 21:02:32.431971 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:32.431821    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4"
Nov 03 21:02:32.431971 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:32.431853    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4" event={"ID":"070c9447-94dc-4af0-90a1-ad328f85bd84","Type":"ContainerDied","Data":"3c9c7cb8c51c1f3f643938ca775e6cfff7b104b236027c262c5ba658cad15bcc"}
Nov 03 21:02:32.431971 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:32.431907    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4" event={"ID":"070c9447-94dc-4af0-90a1-ad328f85bd84","Type":"ContainerDied","Data":"0b52ff34df6e3c391a3c9eb6eda421eda9068abf7ad09f8596c2b413e293958d"}
Nov 03 21:02:32.431971 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:32.431929    4421 scope.go:122] "RemoveContainer" containerID="3c9c7cb8c51c1f3f643938ca775e6cfff7b104b236027c262c5ba658cad15bcc"
Nov 03 21:02:32.442399 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:32.442370    4421 scope.go:122] "RemoveContainer" containerID="3c9c7cb8c51c1f3f643938ca775e6cfff7b104b236027c262c5ba658cad15bcc"
Nov 03 21:02:32.442684 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:02:32.442662    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c9c7cb8c51c1f3f643938ca775e6cfff7b104b236027c262c5ba658cad15bcc\": container with ID starting with 3c9c7cb8c51c1f3f643938ca775e6cfff7b104b236027c262c5ba658cad15bcc not found: ID does not exist" containerID="3c9c7cb8c51c1f3f643938ca775e6cfff7b104b236027c262c5ba658cad15bcc"
Nov 03 21:02:32.442795 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:32.442688    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c9c7cb8c51c1f3f643938ca775e6cfff7b104b236027c262c5ba658cad15bcc"} err="failed to get container status \"3c9c7cb8c51c1f3f643938ca775e6cfff7b104b236027c262c5ba658cad15bcc\": rpc error: code = NotFound desc = could not find container \"3c9c7cb8c51c1f3f643938ca775e6cfff7b104b236027c262c5ba658cad15bcc\": container with ID starting with 3c9c7cb8c51c1f3f643938ca775e6cfff7b104b236027c262c5ba658cad15bcc not found: ID does not exist"
Nov 03 21:02:32.455706 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:32.455677    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4"]
Nov 03 21:02:32.458423 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:32.458407    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b13a14-76fb8586b4-q5tw4"]
Nov 03 21:02:34.015429 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:34.015394    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="070c9447-94dc-4af0-90a1-ad328f85bd84" path="/var/lib/kubelet/pods/070c9447-94dc-4af0-90a1-ad328f85bd84/volumes"
Nov 03 21:02:34.590494 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:34.590458    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th"]
Nov 03 21:02:37.476938 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:37.476815    4421 generic.go:356] "Generic (PLEG): container finished" podID="3a0dff8b-05ed-4261-a30c-08ced3980a2c" containerID="ea0e38b5dc46921faeac45d30eeee38c55748ba3462c9a77155a331e8c5d2b80" exitCode=0
Nov 03 21:02:37.476938 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:37.476943    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq" event={"ID":"3a0dff8b-05ed-4261-a30c-08ced3980a2c","Type":"ContainerDied","Data":"ea0e38b5dc46921faeac45d30eeee38c55748ba3462c9a77155a331e8c5d2b80"}
Nov 03 21:02:37.526645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:37.526623    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq"
Nov 03 21:02:37.646147 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:37.646113    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a0dff8b-05ed-4261-a30c-08ced3980a2c-kube-api-access-sbgbj\" (UniqueName: \"kubernetes.io/projected/3a0dff8b-05ed-4261-a30c-08ced3980a2c-kube-api-access-sbgbj\") pod \"3a0dff8b-05ed-4261-a30c-08ced3980a2c\" (UID: \"3a0dff8b-05ed-4261-a30c-08ced3980a2c\") "
Nov 03 21:02:37.646308 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:37.646240    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a0dff8b-05ed-4261-a30c-08ced3980a2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a0dff8b-05ed-4261-a30c-08ced3980a2c-plugins-config-volume\") pod \"3a0dff8b-05ed-4261-a30c-08ced3980a2c\" (UID: \"3a0dff8b-05ed-4261-a30c-08ced3980a2c\") "
Nov 03 21:02:37.646642 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:37.646584    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a0dff8b-05ed-4261-a30c-08ced3980a2c-plugins-config-volume" pod "3a0dff8b-05ed-4261-a30c-08ced3980a2c" (UID: "3a0dff8b-05ed-4261-a30c-08ced3980a2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:02:37.646878 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:37.646863    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a0dff8b-05ed-4261-a30c-08ced3980a2c-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:37.648518 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:37.648499    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0dff8b-05ed-4261-a30c-08ced3980a2c-kube-api-access-sbgbj" pod "3a0dff8b-05ed-4261-a30c-08ced3980a2c" (UID: "3a0dff8b-05ed-4261-a30c-08ced3980a2c"). InnerVolumeSpecName "kube-api-access-sbgbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:02:37.748151 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:37.748124    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbgbj\" (UniqueName: \"kubernetes.io/projected/3a0dff8b-05ed-4261-a30c-08ced3980a2c-kube-api-access-sbgbj\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:38.489391 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.489362    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq"
Nov 03 21:02:38.489943 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.489400    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq" event={"ID":"3a0dff8b-05ed-4261-a30c-08ced3980a2c","Type":"ContainerDied","Data":"3211fc8c13df7f8e0cac28cc1b8a9df17c1ea1559c9f74060ccbc3c7e7a0e2b6"}
Nov 03 21:02:38.489943 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.489476    4421 scope.go:122] "RemoveContainer" containerID="ea0e38b5dc46921faeac45d30eeee38c55748ba3462c9a77155a331e8c5d2b80"
Nov 03 21:02:38.499119 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.499089    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq" podUID="3a0dff8b-05ed-4261-a30c-08ced3980a2c" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.213:50052\" within 1s: context deadline exceeded"
Nov 03 21:02:38.499239 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:02:38.499173    4421 logging.go:55] [core] [Channel #18598 SubChannel #18599]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.213:50052", ServerName: "10.134.0.213:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.213:50052: operation was canceled"
Nov 03 21:02:38.499239 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.499200    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq"
Nov 03 21:02:38.508545 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.508521    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq"]
Nov 03 21:02:38.515277 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.515258    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-14ebbb-59c4847995-lw7xq"]
Nov 03 21:02:38.547819 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.547788    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t"]
Nov 03 21:02:38.548072 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.548040    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t" podUID="c6e7fd23-0f9d-4ec7-9a57-1326c2a63287" containerName="maas-api" containerID="cri-o://7708efa9313dcf3428eacb3c5b3399cae44f1c2f05bd0366e4e2068548a2a1c1" gracePeriod=30
Nov 03 21:02:38.663686 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.663651    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"]
Nov 03 21:02:38.664496 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.664473    4421 cpu_manager.go:414] "containerMap: removing container" podUID="c70b0ee2-5d82-4575-86ee-a7adc291b6ff" containerName="payload-processing"
Nov 03 21:02:38.664496 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.664496    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c70b0ee2-5d82-4575-86ee-a7adc291b6ff" containerName="payload-processing"
Nov 03 21:02:38.664849 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.664818    4421 cpu_manager.go:414] "containerMap: removing container" podUID="629e965e-1f05-4fdd-a1ef-40aa9da9e3a0" containerName="cleanup"
Nov 03 21:02:38.664849 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.664837    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="629e965e-1f05-4fdd-a1ef-40aa9da9e3a0" containerName="cleanup"
Nov 03 21:02:38.664849 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.664850    4421 cpu_manager.go:414] "containerMap: removing container" podUID="33274f28-f2d3-40fc-a16b-75e90ece007a" containerName="maas-api"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.664858    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33274f28-f2d3-40fc-a16b-75e90ece007a" containerName="maas-api"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.664879    4421 cpu_manager.go:414] "containerMap: removing container" podUID="7caf12c0-4e78-4105-bc0c-7a7e8937e3ac" containerName="cleanup"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.664886    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7caf12c0-4e78-4105-bc0c-7a7e8937e3ac" containerName="cleanup"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.664899    4421 cpu_manager.go:414] "containerMap: removing container" podUID="5107af0d-00d7-415e-be8e-e9ce4d55f8c0" containerName="payload-pre-processing"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.664906    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5107af0d-00d7-415e-be8e-e9ce4d55f8c0" containerName="payload-pre-processing"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.664914    4421 cpu_manager.go:414] "containerMap: removing container" podUID="7caf12c0-4e78-4105-bc0c-7a7e8937e3ac" containerName="cleanup"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.664918    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7caf12c0-4e78-4105-bc0c-7a7e8937e3ac" containerName="cleanup"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.664927    4421 cpu_manager.go:414] "containerMap: removing container" podUID="04160b7a-00d4-47d0-88dd-54646e94e9e5" containerName="payload-processing"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.664931    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04160b7a-00d4-47d0-88dd-54646e94e9e5" containerName="payload-processing"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.664940    4421 cpu_manager.go:414] "containerMap: removing container" podUID="0ae207be-6a8f-4773-bc31-527edb88be5e" containerName="payload-processing"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.664944    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ae207be-6a8f-4773-bc31-527edb88be5e" containerName="payload-processing"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.664956    4421 cpu_manager.go:414] "containerMap: removing container" podUID="070c9447-94dc-4af0-90a1-ad328f85bd84" containerName="maas-api"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.664960    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="070c9447-94dc-4af0-90a1-ad328f85bd84" containerName="maas-api"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.664967    4421 cpu_manager.go:414] "containerMap: removing container" podUID="3b3e4444-3d44-4623-94a7-d4748d93a6e7" containerName="payload-pre-processing"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.664971    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b3e4444-3d44-4623-94a7-d4748d93a6e7" containerName="payload-pre-processing"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.664977    4421 cpu_manager.go:414] "containerMap: removing container" podUID="0a2fda22-a9be-4bf5-8b2b-0e994221c92b" containerName="istio-proxy"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.664981    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a2fda22-a9be-4bf5-8b2b-0e994221c92b" containerName="istio-proxy"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.664992    4421 cpu_manager.go:414] "containerMap: removing container" podUID="68b472a9-c812-4308-895f-fd5193af34f6" containerName="cleanup"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.664996    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68b472a9-c812-4308-895f-fd5193af34f6" containerName="cleanup"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665003    4421 cpu_manager.go:414] "containerMap: removing container" podUID="3a0dff8b-05ed-4261-a30c-08ced3980a2c" containerName="payload-pre-processing"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665007    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a0dff8b-05ed-4261-a30c-08ced3980a2c" containerName="payload-pre-processing"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665015    4421 cpu_manager.go:414] "containerMap: removing container" podUID="eabfcd88-38cc-44ae-bb3c-df42c13f0684" containerName="maas-api"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665020    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eabfcd88-38cc-44ae-bb3c-df42c13f0684" containerName="maas-api"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665037    4421 cpu_manager.go:414] "containerMap: removing container" podUID="bb44d95b-d987-48a4-8220-e1bdf0abfba1" containerName="istio-proxy"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665041    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb44d95b-d987-48a4-8220-e1bdf0abfba1" containerName="istio-proxy"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665049    4421 cpu_manager.go:414] "containerMap: removing container" podUID="4f55c349-f323-4dde-baab-4e279bfcf422" containerName="istio-proxy"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665053    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f55c349-f323-4dde-baab-4e279bfcf422" containerName="istio-proxy"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665063    4421 cpu_manager.go:414] "containerMap: removing container" podUID="4f55c349-f323-4dde-baab-4e279bfcf422" containerName="istio-proxy"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665068    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f55c349-f323-4dde-baab-4e279bfcf422" containerName="istio-proxy"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665087    4421 cpu_manager.go:414] "containerMap: removing container" podUID="629e965e-1f05-4fdd-a1ef-40aa9da9e3a0" containerName="cleanup"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665092    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="629e965e-1f05-4fdd-a1ef-40aa9da9e3a0" containerName="cleanup"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665097    4421 cpu_manager.go:414] "containerMap: removing container" podUID="629e965e-1f05-4fdd-a1ef-40aa9da9e3a0" containerName="cleanup"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665102    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="629e965e-1f05-4fdd-a1ef-40aa9da9e3a0" containerName="cleanup"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665109    4421 cpu_manager.go:414] "containerMap: removing container" podUID="2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8" containerName="payload-processing"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665113    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8" containerName="payload-processing"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665121    4421 cpu_manager.go:414] "containerMap: removing container" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665127    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665134    4421 cpu_manager.go:414] "containerMap: removing container" podUID="0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab" containerName="payload-pre-processing"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665139    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab" containerName="payload-pre-processing"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665146    4421 cpu_manager.go:414] "containerMap: removing container" podUID="38f64ec2-df7a-4c4e-bdd5-d25f13125ad6" containerName="maas-api"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665150    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38f64ec2-df7a-4c4e-bdd5-d25f13125ad6" containerName="maas-api"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665157    4421 cpu_manager.go:414] "containerMap: removing container" podUID="68b472a9-c812-4308-895f-fd5193af34f6" containerName="cleanup"
Nov 03 21:02:38.665203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665161    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68b472a9-c812-4308-895f-fd5193af34f6" containerName="cleanup"
Nov 03 21:02:38.666645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665296    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c70b0ee2-5d82-4575-86ee-a7adc291b6ff" containerName="payload-processing"
Nov 03 21:02:38.666645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665308    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="629e965e-1f05-4fdd-a1ef-40aa9da9e3a0" containerName="cleanup"
Nov 03 21:02:38.666645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665313    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="33274f28-f2d3-40fc-a16b-75e90ece007a" containerName="maas-api"
Nov 03 21:02:38.666645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665319    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="7caf12c0-4e78-4105-bc0c-7a7e8937e3ac" containerName="cleanup"
Nov 03 21:02:38.666645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665324    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="5107af0d-00d7-415e-be8e-e9ce4d55f8c0" containerName="payload-pre-processing"
Nov 03 21:02:38.666645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665330    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="7caf12c0-4e78-4105-bc0c-7a7e8937e3ac" containerName="cleanup"
Nov 03 21:02:38.666645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665335    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="04160b7a-00d4-47d0-88dd-54646e94e9e5" containerName="payload-processing"
Nov 03 21:02:38.666645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665339    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ae207be-6a8f-4773-bc31-527edb88be5e" containerName="payload-processing"
Nov 03 21:02:38.666645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665345    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="070c9447-94dc-4af0-90a1-ad328f85bd84" containerName="maas-api"
Nov 03 21:02:38.666645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665351    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b3e4444-3d44-4623-94a7-d4748d93a6e7" containerName="payload-pre-processing"
Nov 03 21:02:38.666645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665356    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a2fda22-a9be-4bf5-8b2b-0e994221c92b" containerName="istio-proxy"
Nov 03 21:02:38.666645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665361    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="68b472a9-c812-4308-895f-fd5193af34f6" containerName="cleanup"
Nov 03 21:02:38.666645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665366    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a0dff8b-05ed-4261-a30c-08ced3980a2c" containerName="payload-pre-processing"
Nov 03 21:02:38.666645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665372    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="eabfcd88-38cc-44ae-bb3c-df42c13f0684" containerName="maas-api"
Nov 03 21:02:38.666645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665377    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb44d95b-d987-48a4-8220-e1bdf0abfba1" containerName="istio-proxy"
Nov 03 21:02:38.666645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665384    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f55c349-f323-4dde-baab-4e279bfcf422" containerName="istio-proxy"
Nov 03 21:02:38.666645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665390    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f55c349-f323-4dde-baab-4e279bfcf422" containerName="istio-proxy"
Nov 03 21:02:38.666645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665395    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="7caf12c0-4e78-4105-bc0c-7a7e8937e3ac" containerName="cleanup"
Nov 03 21:02:38.666645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665402    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="629e965e-1f05-4fdd-a1ef-40aa9da9e3a0" containerName="cleanup"
Nov 03 21:02:38.666645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665406    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a2fda22-a9be-4bf5-8b2b-0e994221c92b" containerName="istio-proxy"
Nov 03 21:02:38.666645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665412    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="68b472a9-c812-4308-895f-fd5193af34f6" containerName="cleanup"
Nov 03 21:02:38.666645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665419    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f5f33b9-e3df-43c6-bf68-345c3d6ae3b8" containerName="payload-processing"
Nov 03 21:02:38.666645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665427    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7be6b1b-1739-448a-9ca7-27556b447c0f" containerName="main"
Nov 03 21:02:38.666645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665435    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a987b5a-d8ae-43ff-8bf2-b9cfa56916ab" containerName="payload-pre-processing"
Nov 03 21:02:38.666645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665444    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="38f64ec2-df7a-4c4e-bdd5-d25f13125ad6" containerName="maas-api"
Nov 03 21:02:38.666645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.665452    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="68b472a9-c812-4308-895f-fd5193af34f6" containerName="cleanup"
Nov 03 21:02:38.668842 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.668822    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.674993 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.674969    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b13a14-openshift-default-dockercfg-r6zzp\""
Nov 03 21:02:38.686252 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.686225    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"]
Nov 03 21:02:38.799784 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.796272    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db98514-abcc-481f-b53b-e313c205155a-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.799784 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.796413    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.799784 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.796449    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.799784 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.796547    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7bmx\" (UniqueName: \"kubernetes.io/projected/6db98514-abcc-481f-b53b-e313c205155a-kube-api-access-n7bmx\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.799784 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.796575    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db98514-abcc-481f-b53b-e313c205155a-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.799784 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.796628    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.799784 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.796668    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.799784 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.796717    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.799784 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.796783    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db98514-abcc-481f-b53b-e313c205155a-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.900249 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.898038    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.900249 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.898100    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.900249 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.898170    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db98514-abcc-481f-b53b-e313c205155a-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.900249 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.898234    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db98514-abcc-481f-b53b-e313c205155a-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.900249 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.898357    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.900249 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.898392    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.900249 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.898494    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7bmx\" (UniqueName: \"kubernetes.io/projected/6db98514-abcc-481f-b53b-e313c205155a-kube-api-access-n7bmx\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.900249 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.898515    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.900249 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.898527    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db98514-abcc-481f-b53b-e313c205155a-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.900249 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.898573    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.900249 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.898930    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.900249 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.899156    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db98514-abcc-481f-b53b-e313c205155a-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.900249 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.899422    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.900249 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.899476    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.902643 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.901450    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.902643 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.901974    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db98514-abcc-481f-b53b-e313c205155a-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.911122 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.911092    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db98514-abcc-481f-b53b-e313c205155a-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.911513 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.911491    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7bmx\" (UniqueName: \"kubernetes.io/projected/6db98514-abcc-481f-b53b-e313c205155a-kube-api-access-n7bmx\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:38.970004 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.969979    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t"
Nov 03 21:02:38.983288 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:38.983261    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:39.100642 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.100370    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-openshift-service-ca\") pod \"c6e7fd23-0f9d-4ec7-9a57-1326c2a63287\" (UID: \"c6e7fd23-0f9d-4ec7-9a57-1326c2a63287\") "
Nov 03 21:02:39.100642 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.100425    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-kube-api-access-8zv9l\" (UniqueName: \"kubernetes.io/projected/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-kube-api-access-8zv9l\") pod \"c6e7fd23-0f9d-4ec7-9a57-1326c2a63287\" (UID: \"c6e7fd23-0f9d-4ec7-9a57-1326c2a63287\") "
Nov 03 21:02:39.100642 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.100522    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-maas-api-tls\") pod \"c6e7fd23-0f9d-4ec7-9a57-1326c2a63287\" (UID: \"c6e7fd23-0f9d-4ec7-9a57-1326c2a63287\") "
Nov 03 21:02:39.100642 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.100584    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-metrics-certs\") pod \"c6e7fd23-0f9d-4ec7-9a57-1326c2a63287\" (UID: \"c6e7fd23-0f9d-4ec7-9a57-1326c2a63287\") "
Nov 03 21:02:39.101903 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.101871    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-openshift-service-ca" pod "c6e7fd23-0f9d-4ec7-9a57-1326c2a63287" (UID: "c6e7fd23-0f9d-4ec7-9a57-1326c2a63287"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:02:39.103744 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.103713    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-metrics-certs" pod "c6e7fd23-0f9d-4ec7-9a57-1326c2a63287" (UID: "c6e7fd23-0f9d-4ec7-9a57-1326c2a63287"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:02:39.103906 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.103863    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-kube-api-access-8zv9l" pod "c6e7fd23-0f9d-4ec7-9a57-1326c2a63287" (UID: "c6e7fd23-0f9d-4ec7-9a57-1326c2a63287"). InnerVolumeSpecName "kube-api-access-8zv9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:02:39.104442 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.104423    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-maas-api-tls" pod "c6e7fd23-0f9d-4ec7-9a57-1326c2a63287" (UID: "c6e7fd23-0f9d-4ec7-9a57-1326c2a63287"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:02:39.202315 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.202277    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:39.202315 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.202309    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:39.202315 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.202322    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:39.202571 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.202331    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zv9l\" (UniqueName: \"kubernetes.io/projected/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287-kube-api-access-8zv9l\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:39.280120 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.280091    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"]
Nov 03 21:02:39.281477 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:02:39.281446    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db98514_abcc_481f_b53b_e313c205155a.slice/crio-77ad0f492b91fbf31de9b424ee7b98b3b2347a99ec7180991c8efc876bd8e48a WatchSource:0}: Error finding container 77ad0f492b91fbf31de9b424ee7b98b3b2347a99ec7180991c8efc876bd8e48a: Status 404 returned error can't find the container with id 77ad0f492b91fbf31de9b424ee7b98b3b2347a99ec7180991c8efc876bd8e48a
Nov 03 21:02:39.284112 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.284078    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:02:39.284225 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.284146    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:02:39.284225 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.284174    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:02:39.514834 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.514793    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5" event={"ID":"6db98514-abcc-481f-b53b-e313c205155a","Type":"ContainerStarted","Data":"e1fe035a738d06da5ec9232e7443024ca9a1dc499da6312e5ba28b88f928a8b7"}
Nov 03 21:02:39.515313 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.514896    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5" event={"ID":"6db98514-abcc-481f-b53b-e313c205155a","Type":"ContainerStarted","Data":"77ad0f492b91fbf31de9b424ee7b98b3b2347a99ec7180991c8efc876bd8e48a"}
Nov 03 21:02:39.516244 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.516218    4421 generic.go:356] "Generic (PLEG): container finished" podID="c6e7fd23-0f9d-4ec7-9a57-1326c2a63287" containerID="7708efa9313dcf3428eacb3c5b3399cae44f1c2f05bd0366e4e2068548a2a1c1" exitCode=0
Nov 03 21:02:39.516373 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.516273    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t"
Nov 03 21:02:39.516373 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.516278    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t" event={"ID":"c6e7fd23-0f9d-4ec7-9a57-1326c2a63287","Type":"ContainerDied","Data":"7708efa9313dcf3428eacb3c5b3399cae44f1c2f05bd0366e4e2068548a2a1c1"}
Nov 03 21:02:39.516373 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.516323    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t" event={"ID":"c6e7fd23-0f9d-4ec7-9a57-1326c2a63287","Type":"ContainerDied","Data":"0fb8f78f9bfb0a30f9e2bc0657c1e4706c9af086b4f155e24aa0c46f9e77272b"}
Nov 03 21:02:39.516373 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.516351    4421 scope.go:122] "RemoveContainer" containerID="7708efa9313dcf3428eacb3c5b3399cae44f1c2f05bd0366e4e2068548a2a1c1"
Nov 03 21:02:39.528970 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.528939    4421 scope.go:122] "RemoveContainer" containerID="7708efa9313dcf3428eacb3c5b3399cae44f1c2f05bd0366e4e2068548a2a1c1"
Nov 03 21:02:39.529261 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:02:39.529241    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7708efa9313dcf3428eacb3c5b3399cae44f1c2f05bd0366e4e2068548a2a1c1\": container with ID starting with 7708efa9313dcf3428eacb3c5b3399cae44f1c2f05bd0366e4e2068548a2a1c1 not found: ID does not exist" containerID="7708efa9313dcf3428eacb3c5b3399cae44f1c2f05bd0366e4e2068548a2a1c1"
Nov 03 21:02:39.529341 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.529269    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7708efa9313dcf3428eacb3c5b3399cae44f1c2f05bd0366e4e2068548a2a1c1"} err="failed to get container status \"7708efa9313dcf3428eacb3c5b3399cae44f1c2f05bd0366e4e2068548a2a1c1\": rpc error: code = NotFound desc = could not find container \"7708efa9313dcf3428eacb3c5b3399cae44f1c2f05bd0366e4e2068548a2a1c1\": container with ID starting with 7708efa9313dcf3428eacb3c5b3399cae44f1c2f05bd0366e4e2068548a2a1c1 not found: ID does not exist"
Nov 03 21:02:39.541691 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.541638    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5" podStartSLOduration=1.5416212059999999 podStartE2EDuration="1.541621206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:39.537376984 +0000 UTC m=+3689.670505447" watchObservedRunningTime="2026-11-03 21:02:39.541621206 +0000 UTC m=+3689.674749670"
Nov 03 21:02:39.555572 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.555539    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t"]
Nov 03 21:02:39.560128 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.560105    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6350b9-fd64f96dd-9lw8t"]
Nov 03 21:02:39.591793 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.591745    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th" podUID="bcd47c56-cc43-41ff-951b-82c5768ad799" containerName="payload-pre-processing" containerID="cri-o://6f5a5b8322e6b543e74110dad9d2596379d45f9fa68b287df0b3227bfbd53175" gracePeriod=25
Nov 03 21:02:39.984342 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:39.984297    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:40.016560 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:40.016519    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a0dff8b-05ed-4261-a30c-08ced3980a2c" path="/var/lib/kubelet/pods/3a0dff8b-05ed-4261-a30c-08ced3980a2c/volumes"
Nov 03 21:02:40.017098 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:40.017076    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6e7fd23-0f9d-4ec7-9a57-1326c2a63287" path="/var/lib/kubelet/pods/c6e7fd23-0f9d-4ec7-9a57-1326c2a63287/volumes"
Nov 03 21:02:40.985699 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:40.985666    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Nov 03 21:02:40.986099 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:40.985720    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5" podUID="6db98514-abcc-481f-b53b-e313c205155a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Nov 03 21:02:41.983959 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:41.983914    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Nov 03 21:02:41.984194 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:41.983986    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5" podUID="6db98514-abcc-481f-b53b-e313c205155a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Nov 03 21:02:42.984462 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:42.984424    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Nov 03 21:02:42.984871 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:42.984492    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5" podUID="6db98514-abcc-481f-b53b-e313c205155a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Nov 03 21:02:43.077484 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:43.077443    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"]
Nov 03 21:02:43.078371 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:43.078337    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm" podUID="56233474-fdc7-4bcb-aa2a-db87ac074168" containerName="istio-proxy" containerID="cri-o://3c03823108946e2e3da7e4c7ca1035410a393f205ea01f13e56fd81c283fdb92" gracePeriod=30
Nov 03 21:02:43.984144 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:43.984111    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Nov 03 21:02:43.984317 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:43.984166    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5" podUID="6db98514-abcc-481f-b53b-e313c205155a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Nov 03 21:02:44.436974 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:44.436361    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th" podUID="bcd47c56-cc43-41ff-951b-82c5768ad799" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:02:44.984048 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:44.983938    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Nov 03 21:02:44.984048 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:44.984016    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5" podUID="6db98514-abcc-481f-b53b-e313c205155a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Nov 03 21:02:45.984043 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:45.984005    4421 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body=
Nov 03 21:02:45.984542 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:45.984071    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5" podUID="6db98514-abcc-481f-b53b-e313c205155a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused"
Nov 03 21:02:47.208562 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:47.208532    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:47.209353 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:47.209325    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:47.210161 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:47.210142    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:48.333619 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:48.328637    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"]
Nov 03 21:02:48.845898 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:48.845876    4421 generic.go:356] "Generic (PLEG): container finished" podID="56233474-fdc7-4bcb-aa2a-db87ac074168" containerID="3c03823108946e2e3da7e4c7ca1035410a393f205ea01f13e56fd81c283fdb92" exitCode=0
Nov 03 21:02:48.846442 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:48.846415    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm" event={"ID":"56233474-fdc7-4bcb-aa2a-db87ac074168","Type":"ContainerDied","Data":"3c03823108946e2e3da7e4c7ca1035410a393f205ea01f13e56fd81c283fdb92"}
Nov 03 21:02:48.846619 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:48.846577    4421 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5" secret="" err="secret \"e2e-worker-w1-b13a14-openshift-default-dockercfg-r6zzp\" not found"
Nov 03 21:02:48.846851 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:48.846633    4421 scope.go:122] "RemoveContainer" containerID="f509b2406b74172416939d17669932f60e42e4b09fab9ec1df4f36089134c444"
Nov 03 21:02:48.856364 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:48.856344    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 21:02:49.039825 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.039784    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-workload-socket\") pod \"56233474-fdc7-4bcb-aa2a-db87ac074168\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") "
Nov 03 21:02:49.040109 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.039885    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-data\") pod \"56233474-fdc7-4bcb-aa2a-db87ac074168\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") "
Nov 03 21:02:49.040109 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.039987    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-workload-certs\") pod \"56233474-fdc7-4bcb-aa2a-db87ac074168\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") "
Nov 03 21:02:49.040109 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.040026    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-envoy\") pod \"56233474-fdc7-4bcb-aa2a-db87ac074168\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") "
Nov 03 21:02:49.040275 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.040112    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56233474-fdc7-4bcb-aa2a-db87ac074168-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56233474-fdc7-4bcb-aa2a-db87ac074168-istiod-ca-cert\") pod \"56233474-fdc7-4bcb-aa2a-db87ac074168\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") "
Nov 03 21:02:49.040275 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.040119    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-workload-socket" pod "56233474-fdc7-4bcb-aa2a-db87ac074168" (UID: "56233474-fdc7-4bcb-aa2a-db87ac074168"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:02:49.040275 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.040141    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-credential-socket\") pod \"56233474-fdc7-4bcb-aa2a-db87ac074168\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") "
Nov 03 21:02:49.040275 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.040173    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-token\" (UniqueName: \"kubernetes.io/projected/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-token\") pod \"56233474-fdc7-4bcb-aa2a-db87ac074168\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") "
Nov 03 21:02:49.040275 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.040259    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56233474-fdc7-4bcb-aa2a-db87ac074168-kube-api-access-mwkbd\" (UniqueName: \"kubernetes.io/projected/56233474-fdc7-4bcb-aa2a-db87ac074168-kube-api-access-mwkbd\") pod \"56233474-fdc7-4bcb-aa2a-db87ac074168\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") "
Nov 03 21:02:49.040524 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.040302    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-podinfo\") pod \"56233474-fdc7-4bcb-aa2a-db87ac074168\" (UID: \"56233474-fdc7-4bcb-aa2a-db87ac074168\") "
Nov 03 21:02:49.040524 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.040367    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-workload-certs" pod "56233474-fdc7-4bcb-aa2a-db87ac074168" (UID: "56233474-fdc7-4bcb-aa2a-db87ac074168"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:02:49.040524 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.040447    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-data" pod "56233474-fdc7-4bcb-aa2a-db87ac074168" (UID: "56233474-fdc7-4bcb-aa2a-db87ac074168"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:02:49.040703 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.040564    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-credential-socket" pod "56233474-fdc7-4bcb-aa2a-db87ac074168" (UID: "56233474-fdc7-4bcb-aa2a-db87ac074168"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:02:49.042365 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.040905    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:49.042365 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.040930    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:49.042365 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.040944    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:49.042365 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.040958    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:49.042365 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.042273    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56233474-fdc7-4bcb-aa2a-db87ac074168-istiod-ca-cert" pod "56233474-fdc7-4bcb-aa2a-db87ac074168" (UID: "56233474-fdc7-4bcb-aa2a-db87ac074168"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:02:49.043538 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.043512    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-envoy" pod "56233474-fdc7-4bcb-aa2a-db87ac074168" (UID: "56233474-fdc7-4bcb-aa2a-db87ac074168"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:02:49.043710 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.043632    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-token" pod "56233474-fdc7-4bcb-aa2a-db87ac074168" (UID: "56233474-fdc7-4bcb-aa2a-db87ac074168"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:02:49.044083 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.044060    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-podinfo" pod "56233474-fdc7-4bcb-aa2a-db87ac074168" (UID: "56233474-fdc7-4bcb-aa2a-db87ac074168"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:02:49.044631 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.044582    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56233474-fdc7-4bcb-aa2a-db87ac074168-kube-api-access-mwkbd" pod "56233474-fdc7-4bcb-aa2a-db87ac074168" (UID: "56233474-fdc7-4bcb-aa2a-db87ac074168"). InnerVolumeSpecName "kube-api-access-mwkbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:02:49.111775 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.111693    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh"]
Nov 03 21:02:49.143659 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.142733    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56233474-fdc7-4bcb-aa2a-db87ac074168-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:49.143659 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.142763    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:49.143659 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.142778    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwkbd\" (UniqueName: \"kubernetes.io/projected/56233474-fdc7-4bcb-aa2a-db87ac074168-kube-api-access-mwkbd\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:49.143659 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.142791    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:49.143659 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.142804    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56233474-fdc7-4bcb-aa2a-db87ac074168-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:49.653568 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:02:49.653531    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:02:49.854831 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.854804    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"
Nov 03 21:02:49.855003 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.854824    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm" event={"ID":"56233474-fdc7-4bcb-aa2a-db87ac074168","Type":"ContainerDied","Data":"4616839d34d5a9e2ed467d578c6c2a4ca5f4d0f2678bb33ff157677997836fb0"}
Nov 03 21:02:49.855003 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.854923    4421 scope.go:122] "RemoveContainer" containerID="3c03823108946e2e3da7e4c7ca1035410a393f205ea01f13e56fd81c283fdb92"
Nov 03 21:02:49.855143 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.855105    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5" podUID="6db98514-abcc-481f-b53b-e313c205155a" containerName="istio-proxy" containerID="cri-o://e1fe035a738d06da5ec9232e7443024ca9a1dc499da6312e5ba28b88f928a8b7" gracePeriod=30
Nov 03 21:02:49.920334 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.920245    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"]
Nov 03 21:02:49.924733 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:49.924708    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-6350b9-openshift-default-54dcdbf56f-47flm"]
Nov 03 21:02:50.017934 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:50.017901    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56233474-fdc7-4bcb-aa2a-db87ac074168" path="/var/lib/kubelet/pods/56233474-fdc7-4bcb-aa2a-db87ac074168/volumes"
Nov 03 21:02:54.113150 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:54.113090    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh" podUID="6eb20c2e-30b6-4bdd-9b91-fc996180e96d" containerName="payload-pre-processing" containerID="cri-o://a4f46e6818b2d478abea089fd894a83d37fafdddee26a1eddd5941ab4fb2e5d3" gracePeriod=25
Nov 03 21:02:54.411080 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:54.411052    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh"
Nov 03 21:02:54.436064 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:54.436030    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th" podUID="bcd47c56-cc43-41ff-951b-82c5768ad799" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:02:54.543282 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:54.543251    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eb20c2e-30b6-4bdd-9b91-fc996180e96d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb20c2e-30b6-4bdd-9b91-fc996180e96d-plugins-config-volume\") pod \"6eb20c2e-30b6-4bdd-9b91-fc996180e96d\" (UID: \"6eb20c2e-30b6-4bdd-9b91-fc996180e96d\") "
Nov 03 21:02:54.543452 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:54.543433    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eb20c2e-30b6-4bdd-9b91-fc996180e96d-kube-api-access-vs82l\" (UniqueName: \"kubernetes.io/projected/6eb20c2e-30b6-4bdd-9b91-fc996180e96d-kube-api-access-vs82l\") pod \"6eb20c2e-30b6-4bdd-9b91-fc996180e96d\" (UID: \"6eb20c2e-30b6-4bdd-9b91-fc996180e96d\") "
Nov 03 21:02:54.543680 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:54.543654    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eb20c2e-30b6-4bdd-9b91-fc996180e96d-plugins-config-volume" pod "6eb20c2e-30b6-4bdd-9b91-fc996180e96d" (UID: "6eb20c2e-30b6-4bdd-9b91-fc996180e96d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:02:54.543780 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:54.543767    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb20c2e-30b6-4bdd-9b91-fc996180e96d-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:54.545816 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:54.545789    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb20c2e-30b6-4bdd-9b91-fc996180e96d-kube-api-access-vs82l" pod "6eb20c2e-30b6-4bdd-9b91-fc996180e96d" (UID: "6eb20c2e-30b6-4bdd-9b91-fc996180e96d"). InnerVolumeSpecName "kube-api-access-vs82l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:02:54.645080 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:54.644992    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs82l\" (UniqueName: \"kubernetes.io/projected/6eb20c2e-30b6-4bdd-9b91-fc996180e96d-kube-api-access-vs82l\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:54.888948 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:54.888920    4421 generic.go:356] "Generic (PLEG): container finished" podID="6eb20c2e-30b6-4bdd-9b91-fc996180e96d" containerID="a4f46e6818b2d478abea089fd894a83d37fafdddee26a1eddd5941ab4fb2e5d3" exitCode=0
Nov 03 21:02:54.889098 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:54.888985    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh"
Nov 03 21:02:54.889098 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:54.889012    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh" event={"ID":"6eb20c2e-30b6-4bdd-9b91-fc996180e96d","Type":"ContainerDied","Data":"a4f46e6818b2d478abea089fd894a83d37fafdddee26a1eddd5941ab4fb2e5d3"}
Nov 03 21:02:54.889098 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:54.889082    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh" event={"ID":"6eb20c2e-30b6-4bdd-9b91-fc996180e96d","Type":"ContainerDied","Data":"2d0438dc4b22a6067e20bdc7dc4cc707bdfb68ac85e0e42d0928453688d27506"}
Nov 03 21:02:54.889241 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:54.889105    4421 scope.go:122] "RemoveContainer" containerID="a4f46e6818b2d478abea089fd894a83d37fafdddee26a1eddd5941ab4fb2e5d3"
Nov 03 21:02:54.960201 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:54.960172    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh"]
Nov 03 21:02:54.969554 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:54.969524    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bab2c3-54f4bddb5f-lkjxh"]
Nov 03 21:02:54.972389 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:54.972369    4421 scope.go:122] "RemoveContainer" containerID="a4f46e6818b2d478abea089fd894a83d37fafdddee26a1eddd5941ab4fb2e5d3"
Nov 03 21:02:54.972790 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:02:54.972764    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4f46e6818b2d478abea089fd894a83d37fafdddee26a1eddd5941ab4fb2e5d3\": container with ID starting with a4f46e6818b2d478abea089fd894a83d37fafdddee26a1eddd5941ab4fb2e5d3 not found: ID does not exist" containerID="a4f46e6818b2d478abea089fd894a83d37fafdddee26a1eddd5941ab4fb2e5d3"
Nov 03 21:02:54.972875 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:54.972804    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4f46e6818b2d478abea089fd894a83d37fafdddee26a1eddd5941ab4fb2e5d3"} err="failed to get container status \"a4f46e6818b2d478abea089fd894a83d37fafdddee26a1eddd5941ab4fb2e5d3\": rpc error: code = NotFound desc = could not find container \"a4f46e6818b2d478abea089fd894a83d37fafdddee26a1eddd5941ab4fb2e5d3\": container with ID starting with a4f46e6818b2d478abea089fd894a83d37fafdddee26a1eddd5941ab4fb2e5d3 not found: ID does not exist"
Nov 03 21:02:55.112459 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.112434    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:55.255837 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.255741    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-credential-socket\") pod \"6db98514-abcc-481f-b53b-e313c205155a\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") "
Nov 03 21:02:55.255837 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.255804    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db98514-abcc-481f-b53b-e313c205155a-kube-api-access-n7bmx\" (UniqueName: \"kubernetes.io/projected/6db98514-abcc-481f-b53b-e313c205155a-kube-api-access-n7bmx\") pod \"6db98514-abcc-481f-b53b-e313c205155a\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") "
Nov 03 21:02:55.256378 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.255842    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6db98514-abcc-481f-b53b-e313c205155a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db98514-abcc-481f-b53b-e313c205155a-istiod-ca-cert\") pod \"6db98514-abcc-481f-b53b-e313c205155a\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") "
Nov 03 21:02:55.256378 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.255877    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-istio-data\") pod \"6db98514-abcc-481f-b53b-e313c205155a\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") "
Nov 03 21:02:55.256378 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.255970    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-workload-socket\") pod \"6db98514-abcc-481f-b53b-e313c205155a\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") "
Nov 03 21:02:55.256378 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.256072    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db98514-abcc-481f-b53b-e313c205155a-istio-token\" (UniqueName: \"kubernetes.io/projected/6db98514-abcc-481f-b53b-e313c205155a-istio-token\") pod \"6db98514-abcc-481f-b53b-e313c205155a\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") "
Nov 03 21:02:55.256378 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.256095    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-credential-socket" pod "6db98514-abcc-481f-b53b-e313c205155a" (UID: "6db98514-abcc-481f-b53b-e313c205155a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:02:55.256378 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.256149    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-istio-envoy\") pod \"6db98514-abcc-481f-b53b-e313c205155a\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") "
Nov 03 21:02:55.256378 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.256231    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6db98514-abcc-481f-b53b-e313c205155a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db98514-abcc-481f-b53b-e313c205155a-istio-podinfo\") pod \"6db98514-abcc-481f-b53b-e313c205155a\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") "
Nov 03 21:02:55.256378 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.256319    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-workload-certs\") pod \"6db98514-abcc-481f-b53b-e313c205155a\" (UID: \"6db98514-abcc-481f-b53b-e313c205155a\") "
Nov 03 21:02:55.256378 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.256147    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-istio-data" pod "6db98514-abcc-481f-b53b-e313c205155a" (UID: "6db98514-abcc-481f-b53b-e313c205155a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:02:55.256378 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.256344    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-workload-socket" pod "6db98514-abcc-481f-b53b-e313c205155a" (UID: "6db98514-abcc-481f-b53b-e313c205155a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:02:55.257402 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.257240    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:55.257402 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.257266    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:55.257402 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.257302    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:55.257671 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.257493    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-workload-certs" pod "6db98514-abcc-481f-b53b-e313c205155a" (UID: "6db98514-abcc-481f-b53b-e313c205155a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:02:55.257747 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.257731    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db98514-abcc-481f-b53b-e313c205155a-istiod-ca-cert" pod "6db98514-abcc-481f-b53b-e313c205155a" (UID: "6db98514-abcc-481f-b53b-e313c205155a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:02:55.259563 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.258985    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-istio-envoy" pod "6db98514-abcc-481f-b53b-e313c205155a" (UID: "6db98514-abcc-481f-b53b-e313c205155a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:02:55.259563 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.259312    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6db98514-abcc-481f-b53b-e313c205155a-istio-podinfo" pod "6db98514-abcc-481f-b53b-e313c205155a" (UID: "6db98514-abcc-481f-b53b-e313c205155a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:02:55.260263 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.260226    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db98514-abcc-481f-b53b-e313c205155a-kube-api-access-n7bmx" pod "6db98514-abcc-481f-b53b-e313c205155a" (UID: "6db98514-abcc-481f-b53b-e313c205155a"). InnerVolumeSpecName "kube-api-access-n7bmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:02:55.260364 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.260264    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db98514-abcc-481f-b53b-e313c205155a-istio-token" pod "6db98514-abcc-481f-b53b-e313c205155a" (UID: "6db98514-abcc-481f-b53b-e313c205155a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:02:55.358843 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.358808    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db98514-abcc-481f-b53b-e313c205155a-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:55.358843 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.358845    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:55.359059 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.358861    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db98514-abcc-481f-b53b-e313c205155a-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:55.359059 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.358875    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db98514-abcc-481f-b53b-e313c205155a-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:55.359059 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.358890    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7bmx\" (UniqueName: \"kubernetes.io/projected/6db98514-abcc-481f-b53b-e313c205155a-kube-api-access-n7bmx\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:55.359059 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.358903    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db98514-abcc-481f-b53b-e313c205155a-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:02:55.895969 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.895937    4421 generic.go:356] "Generic (PLEG): container finished" podID="6db98514-abcc-481f-b53b-e313c205155a" containerID="e1fe035a738d06da5ec9232e7443024ca9a1dc499da6312e5ba28b88f928a8b7" exitCode=0
Nov 03 21:02:55.896193 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.895982    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"
Nov 03 21:02:55.896193 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.896010    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5" event={"ID":"6db98514-abcc-481f-b53b-e313c205155a","Type":"ContainerDied","Data":"e1fe035a738d06da5ec9232e7443024ca9a1dc499da6312e5ba28b88f928a8b7"}
Nov 03 21:02:55.896193 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.896059    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5" event={"ID":"6db98514-abcc-481f-b53b-e313c205155a","Type":"ContainerDied","Data":"77ad0f492b91fbf31de9b424ee7b98b3b2347a99ec7180991c8efc876bd8e48a"}
Nov 03 21:02:55.896193 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.896082    4421 scope.go:122] "RemoveContainer" containerID="e1fe035a738d06da5ec9232e7443024ca9a1dc499da6312e5ba28b88f928a8b7"
Nov 03 21:02:55.908121 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.908104    4421 scope.go:122] "RemoveContainer" containerID="e1fe035a738d06da5ec9232e7443024ca9a1dc499da6312e5ba28b88f928a8b7"
Nov 03 21:02:55.908425 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:02:55.908395    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1fe035a738d06da5ec9232e7443024ca9a1dc499da6312e5ba28b88f928a8b7\": container with ID starting with e1fe035a738d06da5ec9232e7443024ca9a1dc499da6312e5ba28b88f928a8b7 not found: ID does not exist" containerID="e1fe035a738d06da5ec9232e7443024ca9a1dc499da6312e5ba28b88f928a8b7"
Nov 03 21:02:55.908516 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.908436    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1fe035a738d06da5ec9232e7443024ca9a1dc499da6312e5ba28b88f928a8b7"} err="failed to get container status \"e1fe035a738d06da5ec9232e7443024ca9a1dc499da6312e5ba28b88f928a8b7\": rpc error: code = NotFound desc = could not find container \"e1fe035a738d06da5ec9232e7443024ca9a1dc499da6312e5ba28b88f928a8b7\": container with ID starting with e1fe035a738d06da5ec9232e7443024ca9a1dc499da6312e5ba28b88f928a8b7 not found: ID does not exist"
Nov 03 21:02:55.972686 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.972653    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"]
Nov 03 21:02:55.981620 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:55.977802    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-sssz5"]
Nov 03 21:02:56.017077 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:56.017039    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6db98514-abcc-481f-b53b-e313c205155a" path="/var/lib/kubelet/pods/6db98514-abcc-481f-b53b-e313c205155a/volumes"
Nov 03 21:02:56.017640 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:56.017616    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eb20c2e-30b6-4bdd-9b91-fc996180e96d" path="/var/lib/kubelet/pods/6eb20c2e-30b6-4bdd-9b91-fc996180e96d/volumes"
Nov 03 21:02:59.882275 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:59.882250    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th"
Nov 03 21:02:59.928871 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:59.928841    4421 generic.go:356] "Generic (PLEG): container finished" podID="bcd47c56-cc43-41ff-951b-82c5768ad799" containerID="6f5a5b8322e6b543e74110dad9d2596379d45f9fa68b287df0b3227bfbd53175" exitCode=0
Nov 03 21:02:59.929028 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:59.928902    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th"
Nov 03 21:02:59.929028 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:59.928925    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th" event={"ID":"bcd47c56-cc43-41ff-951b-82c5768ad799","Type":"ContainerDied","Data":"6f5a5b8322e6b543e74110dad9d2596379d45f9fa68b287df0b3227bfbd53175"}
Nov 03 21:02:59.929028 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:59.928993    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th" event={"ID":"bcd47c56-cc43-41ff-951b-82c5768ad799","Type":"ContainerDied","Data":"4b1797ad25af337001a6daa4d42ea511e76629f18cba1563efba9973b95a5d0d"}
Nov 03 21:02:59.929147 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:59.929028    4421 scope.go:122] "RemoveContainer" containerID="6f5a5b8322e6b543e74110dad9d2596379d45f9fa68b287df0b3227bfbd53175"
Nov 03 21:02:59.940846 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:59.940826    4421 scope.go:122] "RemoveContainer" containerID="6f5a5b8322e6b543e74110dad9d2596379d45f9fa68b287df0b3227bfbd53175"
Nov 03 21:02:59.941157 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:02:59.941128    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f5a5b8322e6b543e74110dad9d2596379d45f9fa68b287df0b3227bfbd53175\": container with ID starting with 6f5a5b8322e6b543e74110dad9d2596379d45f9fa68b287df0b3227bfbd53175 not found: ID does not exist" containerID="6f5a5b8322e6b543e74110dad9d2596379d45f9fa68b287df0b3227bfbd53175"
Nov 03 21:02:59.941268 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:02:59.941168    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f5a5b8322e6b543e74110dad9d2596379d45f9fa68b287df0b3227bfbd53175"} err="failed to get container status \"6f5a5b8322e6b543e74110dad9d2596379d45f9fa68b287df0b3227bfbd53175\": rpc error: code = NotFound desc = could not find container \"6f5a5b8322e6b543e74110dad9d2596379d45f9fa68b287df0b3227bfbd53175\": container with ID starting with 6f5a5b8322e6b543e74110dad9d2596379d45f9fa68b287df0b3227bfbd53175 not found: ID does not exist"
Nov 03 21:03:00.031228 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:00.031201    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcd47c56-cc43-41ff-951b-82c5768ad799-kube-api-access-hmcmv\" (UniqueName: \"kubernetes.io/projected/bcd47c56-cc43-41ff-951b-82c5768ad799-kube-api-access-hmcmv\") pod \"bcd47c56-cc43-41ff-951b-82c5768ad799\" (UID: \"bcd47c56-cc43-41ff-951b-82c5768ad799\") "
Nov 03 21:03:00.031422 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:00.031243    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcd47c56-cc43-41ff-951b-82c5768ad799-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcd47c56-cc43-41ff-951b-82c5768ad799-plugins-config-volume\") pod \"bcd47c56-cc43-41ff-951b-82c5768ad799\" (UID: \"bcd47c56-cc43-41ff-951b-82c5768ad799\") "
Nov 03 21:03:00.031679 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:00.031654    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcd47c56-cc43-41ff-951b-82c5768ad799-plugins-config-volume" pod "bcd47c56-cc43-41ff-951b-82c5768ad799" (UID: "bcd47c56-cc43-41ff-951b-82c5768ad799"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:03:00.031896 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:00.031881    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcd47c56-cc43-41ff-951b-82c5768ad799-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:00.033624 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:00.033586    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd47c56-cc43-41ff-951b-82c5768ad799-kube-api-access-hmcmv" pod "bcd47c56-cc43-41ff-951b-82c5768ad799" (UID: "bcd47c56-cc43-41ff-951b-82c5768ad799"). InnerVolumeSpecName "kube-api-access-hmcmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:00.133767 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:00.133693    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmcmv\" (UniqueName: \"kubernetes.io/projected/bcd47c56-cc43-41ff-951b-82c5768ad799-kube-api-access-hmcmv\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:00.265090 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:00.265059    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th"]
Nov 03 21:03:00.272994 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:00.272968    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6350b9-77957cc74c-bp8th"]
Nov 03 21:03:02.016529 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:02.016485    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcd47c56-cc43-41ff-951b-82c5768ad799" path="/var/lib/kubelet/pods/bcd47c56-cc43-41ff-951b-82c5768ad799/volumes"
Nov 03 21:03:05.104793 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:05.104759    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"]
Nov 03 21:03:05.105236 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:05.104994    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6" podUID="ad241f3c-90c0-4f0f-b90c-47eec0af0245" containerName="istio-proxy" containerID="cri-o://ed0605f07816dc22b4488f407845fe8554875e6c77e081ad47f800edb27ac235" gracePeriod=30
Nov 03 21:03:10.364230 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.364203    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 21:03:10.397706 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.394887    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-workload-socket\") pod \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") "
Nov 03 21:03:10.397706 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.394965    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-data\") pod \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") "
Nov 03 21:03:10.397706 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.395021    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-token\" (UniqueName: \"kubernetes.io/projected/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-token\") pod \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") "
Nov 03 21:03:10.397706 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.395109    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-credential-socket\") pod \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") "
Nov 03 21:03:10.397706 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.395152    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-workload-certs\") pod \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") "
Nov 03 21:03:10.397706 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.395204    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad241f3c-90c0-4f0f-b90c-47eec0af0245-kube-api-access-gzt7p\" (UniqueName: \"kubernetes.io/projected/ad241f3c-90c0-4f0f-b90c-47eec0af0245-kube-api-access-gzt7p\") pod \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") "
Nov 03 21:03:10.397706 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.395291    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-podinfo\") pod \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") "
Nov 03 21:03:10.397706 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.395390    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istiod-ca-cert\") pod \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") "
Nov 03 21:03:10.397706 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.395480    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-envoy\") pod \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\" (UID: \"ad241f3c-90c0-4f0f-b90c-47eec0af0245\") "
Nov 03 21:03:10.406466 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.406432    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-data" pod "ad241f3c-90c0-4f0f-b90c-47eec0af0245" (UID: "ad241f3c-90c0-4f0f-b90c-47eec0af0245"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:10.406922 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.406882    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-workload-socket" pod "ad241f3c-90c0-4f0f-b90c-47eec0af0245" (UID: "ad241f3c-90c0-4f0f-b90c-47eec0af0245"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:10.407049 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.407006    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-credential-socket" pod "ad241f3c-90c0-4f0f-b90c-47eec0af0245" (UID: "ad241f3c-90c0-4f0f-b90c-47eec0af0245"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:10.407120 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.407051    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istiod-ca-cert" pod "ad241f3c-90c0-4f0f-b90c-47eec0af0245" (UID: "ad241f3c-90c0-4f0f-b90c-47eec0af0245"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:03:10.407287 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.407255    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-workload-certs" pod "ad241f3c-90c0-4f0f-b90c-47eec0af0245" (UID: "ad241f3c-90c0-4f0f-b90c-47eec0af0245"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:10.409662 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.409636    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-envoy" pod "ad241f3c-90c0-4f0f-b90c-47eec0af0245" (UID: "ad241f3c-90c0-4f0f-b90c-47eec0af0245"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:03:10.409769 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.409684    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad241f3c-90c0-4f0f-b90c-47eec0af0245-kube-api-access-gzt7p" pod "ad241f3c-90c0-4f0f-b90c-47eec0af0245" (UID: "ad241f3c-90c0-4f0f-b90c-47eec0af0245"). InnerVolumeSpecName "kube-api-access-gzt7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:10.412779 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.412752    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-token" pod "ad241f3c-90c0-4f0f-b90c-47eec0af0245" (UID: "ad241f3c-90c0-4f0f-b90c-47eec0af0245"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:10.413073 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.413050    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-podinfo" pod "ad241f3c-90c0-4f0f-b90c-47eec0af0245" (UID: "ad241f3c-90c0-4f0f-b90c-47eec0af0245"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:03:10.497231 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.497194    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:10.497231 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.497226    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:10.497231 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.497236    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:10.497499 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.497245    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzt7p\" (UniqueName: \"kubernetes.io/projected/ad241f3c-90c0-4f0f-b90c-47eec0af0245-kube-api-access-gzt7p\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:10.497499 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.497255    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:10.497499 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.497263    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:10.497499 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.497272    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:10.497499 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.497280    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:10.497499 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.497289    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad241f3c-90c0-4f0f-b90c-47eec0af0245-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:10.997660 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.997612    4421 generic.go:356] "Generic (PLEG): container finished" podID="ad241f3c-90c0-4f0f-b90c-47eec0af0245" containerID="ed0605f07816dc22b4488f407845fe8554875e6c77e081ad47f800edb27ac235" exitCode=0
Nov 03 21:03:10.997861 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.997691    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6" event={"ID":"ad241f3c-90c0-4f0f-b90c-47eec0af0245","Type":"ContainerDied","Data":"ed0605f07816dc22b4488f407845fe8554875e6c77e081ad47f800edb27ac235"}
Nov 03 21:03:10.997861 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.997776    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6" event={"ID":"ad241f3c-90c0-4f0f-b90c-47eec0af0245","Type":"ContainerDied","Data":"89eb04cdf6f1b2552d34e8e41df6b96bd3936bb8295c6f40a6c4576753237cd5"}
Nov 03 21:03:10.997861 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.997697    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"
Nov 03 21:03:10.997861 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:10.997790    4421 scope.go:122] "RemoveContainer" containerID="ed0605f07816dc22b4488f407845fe8554875e6c77e081ad47f800edb27ac235"
Nov 03 21:03:11.008884 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:11.008864    4421 scope.go:122] "RemoveContainer" containerID="ed0605f07816dc22b4488f407845fe8554875e6c77e081ad47f800edb27ac235"
Nov 03 21:03:11.009144 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:03:11.009127    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed0605f07816dc22b4488f407845fe8554875e6c77e081ad47f800edb27ac235\": container with ID starting with ed0605f07816dc22b4488f407845fe8554875e6c77e081ad47f800edb27ac235 not found: ID does not exist" containerID="ed0605f07816dc22b4488f407845fe8554875e6c77e081ad47f800edb27ac235"
Nov 03 21:03:11.009198 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:11.009153    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed0605f07816dc22b4488f407845fe8554875e6c77e081ad47f800edb27ac235"} err="failed to get container status \"ed0605f07816dc22b4488f407845fe8554875e6c77e081ad47f800edb27ac235\": rpc error: code = NotFound desc = could not find container \"ed0605f07816dc22b4488f407845fe8554875e6c77e081ad47f800edb27ac235\": container with ID starting with ed0605f07816dc22b4488f407845fe8554875e6c77e081ad47f800edb27ac235 not found: ID does not exist"
Nov 03 21:03:11.022112 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:11.022086    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"]
Nov 03 21:03:11.026751 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:11.026729    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-kc2h6"]
Nov 03 21:03:12.015272 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:12.015243    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad241f3c-90c0-4f0f-b90c-47eec0af0245" path="/var/lib/kubelet/pods/ad241f3c-90c0-4f0f-b90c-47eec0af0245/volumes"
Nov 03 21:03:19.629652 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:03:19.629548    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:03:33.294649 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:33.294607    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc"]
Nov 03 21:03:34.630961 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:34.630930    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l"]
Nov 03 21:03:34.631445 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:34.631157    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l" podUID="a4b81cef-71a5-4458-97c9-f617fc049a8b" containerName="maas-api" containerID="cri-o://a60569919b5538d4d5a42be5dfd87910d4c0ac0f9351ae673b2cb59e88ba9ea2" gracePeriod=30
Nov 03 21:03:34.958219 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:34.957974    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l"
Nov 03 21:03:35.151535 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:35.151507    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4b81cef-71a5-4458-97c9-f617fc049a8b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4b81cef-71a5-4458-97c9-f617fc049a8b-metrics-certs\") pod \"a4b81cef-71a5-4458-97c9-f617fc049a8b\" (UID: \"a4b81cef-71a5-4458-97c9-f617fc049a8b\") "
Nov 03 21:03:35.151760 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:35.151655    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4b81cef-71a5-4458-97c9-f617fc049a8b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b81cef-71a5-4458-97c9-f617fc049a8b-openshift-service-ca\") pod \"a4b81cef-71a5-4458-97c9-f617fc049a8b\" (UID: \"a4b81cef-71a5-4458-97c9-f617fc049a8b\") "
Nov 03 21:03:35.151872 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:35.151853    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4b81cef-71a5-4458-97c9-f617fc049a8b-kube-api-access-2x8cx\" (UniqueName: \"kubernetes.io/projected/a4b81cef-71a5-4458-97c9-f617fc049a8b-kube-api-access-2x8cx\") pod \"a4b81cef-71a5-4458-97c9-f617fc049a8b\" (UID: \"a4b81cef-71a5-4458-97c9-f617fc049a8b\") "
Nov 03 21:03:35.151941 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:35.151922    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4b81cef-71a5-4458-97c9-f617fc049a8b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b81cef-71a5-4458-97c9-f617fc049a8b-maas-api-tls\") pod \"a4b81cef-71a5-4458-97c9-f617fc049a8b\" (UID: \"a4b81cef-71a5-4458-97c9-f617fc049a8b\") "
Nov 03 21:03:35.152069 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:35.152045    4421 generic.go:356] "Generic (PLEG): container finished" podID="a4b81cef-71a5-4458-97c9-f617fc049a8b" containerID="a60569919b5538d4d5a42be5dfd87910d4c0ac0f9351ae673b2cb59e88ba9ea2" exitCode=0
Nov 03 21:03:35.152130 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:35.152059    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4b81cef-71a5-4458-97c9-f617fc049a8b-openshift-service-ca" pod "a4b81cef-71a5-4458-97c9-f617fc049a8b" (UID: "a4b81cef-71a5-4458-97c9-f617fc049a8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:03:35.152130 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:35.152102    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l"
Nov 03 21:03:35.152214 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:35.152121    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l" event={"ID":"a4b81cef-71a5-4458-97c9-f617fc049a8b","Type":"ContainerDied","Data":"a60569919b5538d4d5a42be5dfd87910d4c0ac0f9351ae673b2cb59e88ba9ea2"}
Nov 03 21:03:35.152214 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:35.152197    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l" event={"ID":"a4b81cef-71a5-4458-97c9-f617fc049a8b","Type":"ContainerDied","Data":"f317fff53e9db684b65008b75eba0f7072cb4a6ed884dd6c52a37438b88e4f21"}
Nov 03 21:03:35.152301 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:35.152232    4421 scope.go:122] "RemoveContainer" containerID="a60569919b5538d4d5a42be5dfd87910d4c0ac0f9351ae673b2cb59e88ba9ea2"
Nov 03 21:03:35.152848 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:35.152784    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4b81cef-71a5-4458-97c9-f617fc049a8b-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:35.154186 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:35.154057    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4b81cef-71a5-4458-97c9-f617fc049a8b-metrics-certs" pod "a4b81cef-71a5-4458-97c9-f617fc049a8b" (UID: "a4b81cef-71a5-4458-97c9-f617fc049a8b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:03:35.154437 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:35.154416    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4b81cef-71a5-4458-97c9-f617fc049a8b-maas-api-tls" pod "a4b81cef-71a5-4458-97c9-f617fc049a8b" (UID: "a4b81cef-71a5-4458-97c9-f617fc049a8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:03:35.154669 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:35.154651    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b81cef-71a5-4458-97c9-f617fc049a8b-kube-api-access-2x8cx" pod "a4b81cef-71a5-4458-97c9-f617fc049a8b" (UID: "a4b81cef-71a5-4458-97c9-f617fc049a8b"). InnerVolumeSpecName "kube-api-access-2x8cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:35.177708 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:35.177672    4421 scope.go:122] "RemoveContainer" containerID="a60569919b5538d4d5a42be5dfd87910d4c0ac0f9351ae673b2cb59e88ba9ea2"
Nov 03 21:03:35.177983 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:03:35.177957    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a60569919b5538d4d5a42be5dfd87910d4c0ac0f9351ae673b2cb59e88ba9ea2\": container with ID starting with a60569919b5538d4d5a42be5dfd87910d4c0ac0f9351ae673b2cb59e88ba9ea2 not found: ID does not exist" containerID="a60569919b5538d4d5a42be5dfd87910d4c0ac0f9351ae673b2cb59e88ba9ea2"
Nov 03 21:03:35.178087 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:35.177989    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a60569919b5538d4d5a42be5dfd87910d4c0ac0f9351ae673b2cb59e88ba9ea2"} err="failed to get container status \"a60569919b5538d4d5a42be5dfd87910d4c0ac0f9351ae673b2cb59e88ba9ea2\": rpc error: code = NotFound desc = could not find container \"a60569919b5538d4d5a42be5dfd87910d4c0ac0f9351ae673b2cb59e88ba9ea2\": container with ID starting with a60569919b5538d4d5a42be5dfd87910d4c0ac0f9351ae673b2cb59e88ba9ea2 not found: ID does not exist"
Nov 03 21:03:35.254263 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:35.254234    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x8cx\" (UniqueName: \"kubernetes.io/projected/a4b81cef-71a5-4458-97c9-f617fc049a8b-kube-api-access-2x8cx\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:35.254263 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:35.254262    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4b81cef-71a5-4458-97c9-f617fc049a8b-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:35.254486 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:35.254274    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4b81cef-71a5-4458-97c9-f617fc049a8b-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:35.475738 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:35.475707    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l"]
Nov 03 21:03:35.479216 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:35.479194    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3bcc2b-8645588fb4-68t5l"]
Nov 03 21:03:36.025332 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:36.025294    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4b81cef-71a5-4458-97c9-f617fc049a8b" path="/var/lib/kubelet/pods/a4b81cef-71a5-4458-97c9-f617fc049a8b/volumes"
Nov 03 21:03:38.295189 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:38.295140    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc" podUID="feb6a504-7873-440d-a013-36654e702307" containerName="payload-processing" containerID="cri-o://b061e3e4a0f02d7919a177b0ecba3e0270825a1ecafaa442df0a21d23bd11cb7" gracePeriod=25
Nov 03 21:03:38.566391 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:38.566368    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc"
Nov 03 21:03:38.593261 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:38.593226    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feb6a504-7873-440d-a013-36654e702307-kube-api-access-2wjsr\" (UniqueName: \"kubernetes.io/projected/feb6a504-7873-440d-a013-36654e702307-kube-api-access-2wjsr\") pod \"feb6a504-7873-440d-a013-36654e702307\" (UID: \"feb6a504-7873-440d-a013-36654e702307\") "
Nov 03 21:03:38.593449 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:38.593288    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feb6a504-7873-440d-a013-36654e702307-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb6a504-7873-440d-a013-36654e702307-plugins-config-volume\") pod \"feb6a504-7873-440d-a013-36654e702307\" (UID: \"feb6a504-7873-440d-a013-36654e702307\") "
Nov 03 21:03:38.594425 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:38.593778    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feb6a504-7873-440d-a013-36654e702307-plugins-config-volume" pod "feb6a504-7873-440d-a013-36654e702307" (UID: "feb6a504-7873-440d-a013-36654e702307"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:03:38.594425 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:38.593997    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feb6a504-7873-440d-a013-36654e702307-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:38.596756 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:38.596726    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb6a504-7873-440d-a013-36654e702307-kube-api-access-2wjsr" pod "feb6a504-7873-440d-a013-36654e702307" (UID: "feb6a504-7873-440d-a013-36654e702307"). InnerVolumeSpecName "kube-api-access-2wjsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:03:38.695891 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:38.695856    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wjsr\" (UniqueName: \"kubernetes.io/projected/feb6a504-7873-440d-a013-36654e702307-kube-api-access-2wjsr\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:03:39.181886 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:39.181850    4421 generic.go:356] "Generic (PLEG): container finished" podID="feb6a504-7873-440d-a013-36654e702307" containerID="b061e3e4a0f02d7919a177b0ecba3e0270825a1ecafaa442df0a21d23bd11cb7" exitCode=0
Nov 03 21:03:39.182074 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:39.181900    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc"
Nov 03 21:03:39.182074 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:39.181927    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc" event={"ID":"feb6a504-7873-440d-a013-36654e702307","Type":"ContainerDied","Data":"b061e3e4a0f02d7919a177b0ecba3e0270825a1ecafaa442df0a21d23bd11cb7"}
Nov 03 21:03:39.182074 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:39.181979    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc" event={"ID":"feb6a504-7873-440d-a013-36654e702307","Type":"ContainerDied","Data":"a033bfe1e3d51daed22dec547fa78b98df9b6d9ae395af6b2536551fa52554a9"}
Nov 03 21:03:39.182074 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:39.182008    4421 scope.go:122] "RemoveContainer" containerID="b061e3e4a0f02d7919a177b0ecba3e0270825a1ecafaa442df0a21d23bd11cb7"
Nov 03 21:03:39.192901 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:39.192882    4421 scope.go:122] "RemoveContainer" containerID="b061e3e4a0f02d7919a177b0ecba3e0270825a1ecafaa442df0a21d23bd11cb7"
Nov 03 21:03:39.193182 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:03:39.193161    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b061e3e4a0f02d7919a177b0ecba3e0270825a1ecafaa442df0a21d23bd11cb7\": container with ID starting with b061e3e4a0f02d7919a177b0ecba3e0270825a1ecafaa442df0a21d23bd11cb7 not found: ID does not exist" containerID="b061e3e4a0f02d7919a177b0ecba3e0270825a1ecafaa442df0a21d23bd11cb7"
Nov 03 21:03:39.193260 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:39.193192    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b061e3e4a0f02d7919a177b0ecba3e0270825a1ecafaa442df0a21d23bd11cb7"} err="failed to get container status \"b061e3e4a0f02d7919a177b0ecba3e0270825a1ecafaa442df0a21d23bd11cb7\": rpc error: code = NotFound desc = could not find container \"b061e3e4a0f02d7919a177b0ecba3e0270825a1ecafaa442df0a21d23bd11cb7\": container with ID starting with b061e3e4a0f02d7919a177b0ecba3e0270825a1ecafaa442df0a21d23bd11cb7 not found: ID does not exist"
Nov 03 21:03:39.205839 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:39.205817    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc"]
Nov 03 21:03:39.209384 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:39.209357    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-4f93d1-79f8f9b54b-7j9mc"]
Nov 03 21:03:40.016844 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:03:40.016802    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feb6a504-7873-440d-a013-36654e702307" path="/var/lib/kubelet/pods/feb6a504-7873-440d-a013-36654e702307/volumes"
Nov 03 21:03:49.632093 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:03:49.632051    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:04:00.065056 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:00.065012    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m"]
Nov 03 21:04:03.095642 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:03.095586    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"]
Nov 03 21:04:03.096158 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:03.095807    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd" podUID="657622a4-6298-4496-b2c3-edce95190d04" containerName="istio-proxy" containerID="cri-o://b782e8874dc9f21a856019f11c2650c2ae2f54b007300ca1d318df51d801f9d2" gracePeriod=30
Nov 03 21:04:05.066302 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:05.066265    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m" podUID="5b495a05-8e6f-4b43-b9d3-20ad6b499731" containerName="payload-processing" containerID="cri-o://0cab1cf353d3092be16a0075f6a6582051379fedad8f3b5139e7f3ef0a3c043f" gracePeriod=25
Nov 03 21:04:05.369122 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:05.369092    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m"
Nov 03 21:04:05.369880 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:05.369852    4421 generic.go:356] "Generic (PLEG): container finished" podID="5b495a05-8e6f-4b43-b9d3-20ad6b499731" containerID="0cab1cf353d3092be16a0075f6a6582051379fedad8f3b5139e7f3ef0a3c043f" exitCode=0
Nov 03 21:04:05.370045 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:05.369887    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m" event={"ID":"5b495a05-8e6f-4b43-b9d3-20ad6b499731","Type":"ContainerDied","Data":"0cab1cf353d3092be16a0075f6a6582051379fedad8f3b5139e7f3ef0a3c043f"}
Nov 03 21:04:05.370045 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:05.369928    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m" event={"ID":"5b495a05-8e6f-4b43-b9d3-20ad6b499731","Type":"ContainerDied","Data":"b240b885cca00d87ef5428e02504df2623bb657c39377294d0366c040c15533e"}
Nov 03 21:04:05.370045 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:05.369949    4421 scope.go:122] "RemoveContainer" containerID="0cab1cf353d3092be16a0075f6a6582051379fedad8f3b5139e7f3ef0a3c043f"
Nov 03 21:04:05.379937 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:05.379920    4421 scope.go:122] "RemoveContainer" containerID="0cab1cf353d3092be16a0075f6a6582051379fedad8f3b5139e7f3ef0a3c043f"
Nov 03 21:04:05.380209 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:04:05.380191    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cab1cf353d3092be16a0075f6a6582051379fedad8f3b5139e7f3ef0a3c043f\": container with ID starting with 0cab1cf353d3092be16a0075f6a6582051379fedad8f3b5139e7f3ef0a3c043f not found: ID does not exist" containerID="0cab1cf353d3092be16a0075f6a6582051379fedad8f3b5139e7f3ef0a3c043f"
Nov 03 21:04:05.380268 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:05.380215    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cab1cf353d3092be16a0075f6a6582051379fedad8f3b5139e7f3ef0a3c043f"} err="failed to get container status \"0cab1cf353d3092be16a0075f6a6582051379fedad8f3b5139e7f3ef0a3c043f\": rpc error: code = NotFound desc = could not find container \"0cab1cf353d3092be16a0075f6a6582051379fedad8f3b5139e7f3ef0a3c043f\": container with ID starting with 0cab1cf353d3092be16a0075f6a6582051379fedad8f3b5139e7f3ef0a3c043f not found: ID does not exist"
Nov 03 21:04:05.495437 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:05.495406    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b495a05-8e6f-4b43-b9d3-20ad6b499731-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b495a05-8e6f-4b43-b9d3-20ad6b499731-plugins-config-volume\") pod \"5b495a05-8e6f-4b43-b9d3-20ad6b499731\" (UID: \"5b495a05-8e6f-4b43-b9d3-20ad6b499731\") "
Nov 03 21:04:05.495639 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:05.495480    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b495a05-8e6f-4b43-b9d3-20ad6b499731-kube-api-access-jhpj6\" (UniqueName: \"kubernetes.io/projected/5b495a05-8e6f-4b43-b9d3-20ad6b499731-kube-api-access-jhpj6\") pod \"5b495a05-8e6f-4b43-b9d3-20ad6b499731\" (UID: \"5b495a05-8e6f-4b43-b9d3-20ad6b499731\") "
Nov 03 21:04:05.495898 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:05.495867    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b495a05-8e6f-4b43-b9d3-20ad6b499731-plugins-config-volume" pod "5b495a05-8e6f-4b43-b9d3-20ad6b499731" (UID: "5b495a05-8e6f-4b43-b9d3-20ad6b499731"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:05.496184 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:05.496169    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b495a05-8e6f-4b43-b9d3-20ad6b499731-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:05.497747 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:05.497719    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b495a05-8e6f-4b43-b9d3-20ad6b499731-kube-api-access-jhpj6" pod "5b495a05-8e6f-4b43-b9d3-20ad6b499731" (UID: "5b495a05-8e6f-4b43-b9d3-20ad6b499731"). InnerVolumeSpecName "kube-api-access-jhpj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:05.597344 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:05.597258    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhpj6\" (UniqueName: \"kubernetes.io/projected/5b495a05-8e6f-4b43-b9d3-20ad6b499731-kube-api-access-jhpj6\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:06.019043 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:06.019012    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"]
Nov 03 21:04:06.019302 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:06.019272    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d" podUID="c9adc7d2-3c5b-42e1-b790-20db31c15593" containerName="istio-proxy" containerID="cri-o://8b5c105b65710bcd692af2980de1b6e91d53e47ae87cbd94742b3a21ba0d81cd" gracePeriod=30
Nov 03 21:04:06.375970 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:06.375893    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m"
Nov 03 21:04:06.395704 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:06.395674    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m"]
Nov 03 21:04:06.401126 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:06.401099    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-c7aecb-5974bf759b-ggb9m"]
Nov 03 21:04:07.271027 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:07.270993    4421 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:04:07.271227 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:07.271051    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d" podUID="c9adc7d2-3c5b-42e1-b790-20db31c15593" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:04:07.918750 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:07.918712    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-w9chd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:04:07.919160 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:07.918773    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd" podUID="657622a4-6298-4496-b2c3-edce95190d04" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:04:08.015345 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.015307    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b495a05-8e6f-4b43-b9d3-20ad6b499731" path="/var/lib/kubelet/pods/5b495a05-8e6f-4b43-b9d3-20ad6b499731/volumes"
Nov 03 21:04:08.388397 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.388374    4421 generic.go:356] "Generic (PLEG): container finished" podID="657622a4-6298-4496-b2c3-edce95190d04" containerID="b782e8874dc9f21a856019f11c2650c2ae2f54b007300ca1d318df51d801f9d2" exitCode=0
Nov 03 21:04:08.388568 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.388454    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd" event={"ID":"657622a4-6298-4496-b2c3-edce95190d04","Type":"ContainerDied","Data":"b782e8874dc9f21a856019f11c2650c2ae2f54b007300ca1d318df51d801f9d2"}
Nov 03 21:04:08.388568 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.388512    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd" event={"ID":"657622a4-6298-4496-b2c3-edce95190d04","Type":"ContainerDied","Data":"d890e08fb82c5b58c3e22915fd3e75da349ca675c72a133458f422eb9ee4e558"}
Nov 03 21:04:08.388568 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.388537    4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d890e08fb82c5b58c3e22915fd3e75da349ca675c72a133458f422eb9ee4e558"
Nov 03 21:04:08.390026 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.390005    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 21:04:08.436042 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.436007    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-credential-socket\") pod \"657622a4-6298-4496-b2c3-edce95190d04\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") "
Nov 03 21:04:08.436219 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.436067    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-workload-certs\") pod \"657622a4-6298-4496-b2c3-edce95190d04\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") "
Nov 03 21:04:08.436219 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.436098    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-workload-socket\") pod \"657622a4-6298-4496-b2c3-edce95190d04\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") "
Nov 03 21:04:08.436219 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.436167    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/657622a4-6298-4496-b2c3-edce95190d04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657622a4-6298-4496-b2c3-edce95190d04-istio-podinfo\") pod \"657622a4-6298-4496-b2c3-edce95190d04\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") "
Nov 03 21:04:08.436416 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.436229    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-istio-envoy\") pod \"657622a4-6298-4496-b2c3-edce95190d04\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") "
Nov 03 21:04:08.436416 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.436276    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/657622a4-6298-4496-b2c3-edce95190d04-kube-api-access-dxh57\" (UniqueName: \"kubernetes.io/projected/657622a4-6298-4496-b2c3-edce95190d04-kube-api-access-dxh57\") pod \"657622a4-6298-4496-b2c3-edce95190d04\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") "
Nov 03 21:04:08.436416 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.436350    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-credential-socket" pod "657622a4-6298-4496-b2c3-edce95190d04" (UID: "657622a4-6298-4496-b2c3-edce95190d04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:08.436587 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.436412    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-workload-certs" pod "657622a4-6298-4496-b2c3-edce95190d04" (UID: "657622a4-6298-4496-b2c3-edce95190d04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:08.436587 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.436425    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/657622a4-6298-4496-b2c3-edce95190d04-istio-token\" (UniqueName: \"kubernetes.io/projected/657622a4-6298-4496-b2c3-edce95190d04-istio-token\") pod \"657622a4-6298-4496-b2c3-edce95190d04\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") "
Nov 03 21:04:08.436587 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.436487    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/657622a4-6298-4496-b2c3-edce95190d04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657622a4-6298-4496-b2c3-edce95190d04-istiod-ca-cert\") pod \"657622a4-6298-4496-b2c3-edce95190d04\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") "
Nov 03 21:04:08.436587 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.436541    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-istio-data\") pod \"657622a4-6298-4496-b2c3-edce95190d04\" (UID: \"657622a4-6298-4496-b2c3-edce95190d04\") "
Nov 03 21:04:08.436587 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.436553    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-workload-socket" pod "657622a4-6298-4496-b2c3-edce95190d04" (UID: "657622a4-6298-4496-b2c3-edce95190d04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:08.437142 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.437067    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/657622a4-6298-4496-b2c3-edce95190d04-istiod-ca-cert" pod "657622a4-6298-4496-b2c3-edce95190d04" (UID: "657622a4-6298-4496-b2c3-edce95190d04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:08.437243 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.437089    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:08.437332 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.437310    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-istio-data" pod "657622a4-6298-4496-b2c3-edce95190d04" (UID: "657622a4-6298-4496-b2c3-edce95190d04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:08.437397 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.437357    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:08.437397 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.437378    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:08.439436 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.439397    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-istio-envoy" pod "657622a4-6298-4496-b2c3-edce95190d04" (UID: "657622a4-6298-4496-b2c3-edce95190d04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:08.439436 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.439424    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657622a4-6298-4496-b2c3-edce95190d04-kube-api-access-dxh57" pod "657622a4-6298-4496-b2c3-edce95190d04" (UID: "657622a4-6298-4496-b2c3-edce95190d04"). InnerVolumeSpecName "kube-api-access-dxh57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:08.439585 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.439453    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657622a4-6298-4496-b2c3-edce95190d04-istio-token" pod "657622a4-6298-4496-b2c3-edce95190d04" (UID: "657622a4-6298-4496-b2c3-edce95190d04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:08.439585 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.439483    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/657622a4-6298-4496-b2c3-edce95190d04-istio-podinfo" pod "657622a4-6298-4496-b2c3-edce95190d04" (UID: "657622a4-6298-4496-b2c3-edce95190d04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:04:08.538534 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.538457    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:08.538534 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.538487    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/657622a4-6298-4496-b2c3-edce95190d04-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:08.538534 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.538498    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/657622a4-6298-4496-b2c3-edce95190d04-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:08.538534 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.538507    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxh57\" (UniqueName: \"kubernetes.io/projected/657622a4-6298-4496-b2c3-edce95190d04-kube-api-access-dxh57\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:08.538534 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.538517    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/657622a4-6298-4496-b2c3-edce95190d04-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:08.538534 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:08.538525    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/657622a4-6298-4496-b2c3-edce95190d04-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:09.394031 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:09.393999    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"
Nov 03 21:04:09.442353 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:09.442322    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"]
Nov 03 21:04:09.452282 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:09.452257    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-w9chd"]
Nov 03 21:04:09.940158 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:09.940122    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"]
Nov 03 21:04:09.940385 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:09.940360    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm" podUID="b8cf0dee-6ac9-4932-9f4c-0d475d4597ae" containerName="istio-proxy" containerID="cri-o://ebf0b18190f0973556f120eebc7cc6284a7be162e983760cc68a4d1232324eb6" gracePeriod=30
Nov 03 21:04:10.017250 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:10.017217    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="657622a4-6298-4496-b2c3-edce95190d04" path="/var/lib/kubelet/pods/657622a4-6298-4496-b2c3-edce95190d04/volumes"
Nov 03 21:04:11.293524 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.293466    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 21:04:11.387381 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.387347    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-token\" (UniqueName: \"kubernetes.io/projected/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-token\") pod \"c9adc7d2-3c5b-42e1-b790-20db31c15593\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") "
Nov 03 21:04:11.387561 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.387388    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-credential-socket\") pod \"c9adc7d2-3c5b-42e1-b790-20db31c15593\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") "
Nov 03 21:04:11.387561 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.387432    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9adc7d2-3c5b-42e1-b790-20db31c15593-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9adc7d2-3c5b-42e1-b790-20db31c15593-istiod-ca-cert\") pod \"c9adc7d2-3c5b-42e1-b790-20db31c15593\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") "
Nov 03 21:04:11.387561 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.387512    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-data\") pod \"c9adc7d2-3c5b-42e1-b790-20db31c15593\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") "
Nov 03 21:04:11.387770 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.387696    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-workload-certs\") pod \"c9adc7d2-3c5b-42e1-b790-20db31c15593\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") "
Nov 03 21:04:11.387770 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.387758    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9adc7d2-3c5b-42e1-b790-20db31c15593-kube-api-access-7j87d\" (UniqueName: \"kubernetes.io/projected/c9adc7d2-3c5b-42e1-b790-20db31c15593-kube-api-access-7j87d\") pod \"c9adc7d2-3c5b-42e1-b790-20db31c15593\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") "
Nov 03 21:04:11.387881 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.387798    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-envoy\") pod \"c9adc7d2-3c5b-42e1-b790-20db31c15593\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") "
Nov 03 21:04:11.387881 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.387832    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-credential-socket" pod "c9adc7d2-3c5b-42e1-b790-20db31c15593" (UID: "c9adc7d2-3c5b-42e1-b790-20db31c15593"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:11.387881 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.387848    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-workload-socket\") pod \"c9adc7d2-3c5b-42e1-b790-20db31c15593\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") "
Nov 03 21:04:11.388046 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.387891    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-data" pod "c9adc7d2-3c5b-42e1-b790-20db31c15593" (UID: "c9adc7d2-3c5b-42e1-b790-20db31c15593"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:11.388046 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.387936    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-podinfo\") pod \"c9adc7d2-3c5b-42e1-b790-20db31c15593\" (UID: \"c9adc7d2-3c5b-42e1-b790-20db31c15593\") "
Nov 03 21:04:11.388046 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.387954    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-workload-certs" pod "c9adc7d2-3c5b-42e1-b790-20db31c15593" (UID: "c9adc7d2-3c5b-42e1-b790-20db31c15593"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:11.388046 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.387975    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9adc7d2-3c5b-42e1-b790-20db31c15593-istiod-ca-cert" pod "c9adc7d2-3c5b-42e1-b790-20db31c15593" (UID: "c9adc7d2-3c5b-42e1-b790-20db31c15593"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:11.388237 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.388094    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-workload-socket" pod "c9adc7d2-3c5b-42e1-b790-20db31c15593" (UID: "c9adc7d2-3c5b-42e1-b790-20db31c15593"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:11.388659 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.388633    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:11.388800 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.388664    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:11.388800 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.388680    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:11.388800 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.388695    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9adc7d2-3c5b-42e1-b790-20db31c15593-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:11.388800 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.388709    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:11.389869 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.389847    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-token" pod "c9adc7d2-3c5b-42e1-b790-20db31c15593" (UID: "c9adc7d2-3c5b-42e1-b790-20db31c15593"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:11.390155 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.390134    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9adc7d2-3c5b-42e1-b790-20db31c15593-kube-api-access-7j87d" pod "c9adc7d2-3c5b-42e1-b790-20db31c15593" (UID: "c9adc7d2-3c5b-42e1-b790-20db31c15593"). InnerVolumeSpecName "kube-api-access-7j87d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:11.390239 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.390224    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-envoy" pod "c9adc7d2-3c5b-42e1-b790-20db31c15593" (UID: "c9adc7d2-3c5b-42e1-b790-20db31c15593"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:11.390332 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.390313    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-podinfo" pod "c9adc7d2-3c5b-42e1-b790-20db31c15593" (UID: "c9adc7d2-3c5b-42e1-b790-20db31c15593"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:04:11.406637 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.406576    4421 generic.go:356] "Generic (PLEG): container finished" podID="c9adc7d2-3c5b-42e1-b790-20db31c15593" containerID="8b5c105b65710bcd692af2980de1b6e91d53e47ae87cbd94742b3a21ba0d81cd" exitCode=0
Nov 03 21:04:11.406756 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.406632    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d" event={"ID":"c9adc7d2-3c5b-42e1-b790-20db31c15593","Type":"ContainerDied","Data":"8b5c105b65710bcd692af2980de1b6e91d53e47ae87cbd94742b3a21ba0d81cd"}
Nov 03 21:04:11.406756 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.406656    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"
Nov 03 21:04:11.406756 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.406692    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d" event={"ID":"c9adc7d2-3c5b-42e1-b790-20db31c15593","Type":"ContainerDied","Data":"6d34843ec925038aa75fdd494bd40c0ca4be42d67c37b4935cd05d4cd958f4cc"}
Nov 03 21:04:11.406756 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.406712    4421 scope.go:122] "RemoveContainer" containerID="8b5c105b65710bcd692af2980de1b6e91d53e47ae87cbd94742b3a21ba0d81cd"
Nov 03 21:04:11.416972 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.416958    4421 scope.go:122] "RemoveContainer" containerID="8b5c105b65710bcd692af2980de1b6e91d53e47ae87cbd94742b3a21ba0d81cd"
Nov 03 21:04:11.417266 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:04:11.417244    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b5c105b65710bcd692af2980de1b6e91d53e47ae87cbd94742b3a21ba0d81cd\": container with ID starting with 8b5c105b65710bcd692af2980de1b6e91d53e47ae87cbd94742b3a21ba0d81cd not found: ID does not exist" containerID="8b5c105b65710bcd692af2980de1b6e91d53e47ae87cbd94742b3a21ba0d81cd"
Nov 03 21:04:11.417325 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.417278    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b5c105b65710bcd692af2980de1b6e91d53e47ae87cbd94742b3a21ba0d81cd"} err="failed to get container status \"8b5c105b65710bcd692af2980de1b6e91d53e47ae87cbd94742b3a21ba0d81cd\": rpc error: code = NotFound desc = could not find container \"8b5c105b65710bcd692af2980de1b6e91d53e47ae87cbd94742b3a21ba0d81cd\": container with ID starting with 8b5c105b65710bcd692af2980de1b6e91d53e47ae87cbd94742b3a21ba0d81cd not found: ID does not exist"
Nov 03 21:04:11.432546 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.432515    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"]
Nov 03 21:04:11.439654 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.439628    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-9dm5d"]
Nov 03 21:04:11.489386 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.489353    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:11.489386 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.489381    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:11.489386 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.489392    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9adc7d2-3c5b-42e1-b790-20db31c15593-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:11.489644 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:11.489404    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j87d\" (UniqueName: \"kubernetes.io/projected/c9adc7d2-3c5b-42e1-b790-20db31c15593-kube-api-access-7j87d\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:12.017136 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:12.017100    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9adc7d2-3c5b-42e1-b790-20db31c15593" path="/var/lib/kubelet/pods/c9adc7d2-3c5b-42e1-b790-20db31c15593/volumes"
Nov 03 21:04:13.522806 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:13.522780    4421 scope.go:122] "RemoveContainer" containerID="b782e8874dc9f21a856019f11c2650c2ae2f54b007300ca1d318df51d801f9d2"
Nov 03 21:04:15.254866 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.254841    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 21:04:15.445236 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.445203    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"]
Nov 03 21:04:15.445886 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.445847    4421 generic.go:356] "Generic (PLEG): container finished" podID="b8cf0dee-6ac9-4932-9f4c-0d475d4597ae" containerID="ebf0b18190f0973556f120eebc7cc6284a7be162e983760cc68a4d1232324eb6" exitCode=0
Nov 03 21:04:15.446024 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.445886    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"
Nov 03 21:04:15.446024 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.445924    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm" event={"ID":"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae","Type":"ContainerDied","Data":"ebf0b18190f0973556f120eebc7cc6284a7be162e983760cc68a4d1232324eb6"}
Nov 03 21:04:15.446024 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.445996    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm" event={"ID":"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae","Type":"ContainerDied","Data":"5c2e35e7f96ffc2e84aeca33016304c74685c02546ebffd215972a08ddf72fe9"}
Nov 03 21:04:15.446024 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.446018    4421 scope.go:122] "RemoveContainer" containerID="ebf0b18190f0973556f120eebc7cc6284a7be162e983760cc68a4d1232324eb6"
Nov 03 21:04:15.446805 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.446786    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istiod-ca-cert\") pod \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") "
Nov 03 21:04:15.446943 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.446834    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-workload-certs\") pod \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") "
Nov 03 21:04:15.446943 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.446907    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-token\" (UniqueName: \"kubernetes.io/projected/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-token\") pod \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") "
Nov 03 21:04:15.447066 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.446950    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-data\") pod \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") "
Nov 03 21:04:15.447202 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.447132    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-workload-certs" pod "b8cf0dee-6ac9-4932-9f4c-0d475d4597ae" (UID: "b8cf0dee-6ac9-4932-9f4c-0d475d4597ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:15.447202 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.447141    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-envoy\") pod \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") "
Nov 03 21:04:15.447202 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.447168    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istiod-ca-cert" pod "b8cf0dee-6ac9-4932-9f4c-0d475d4597ae" (UID: "b8cf0dee-6ac9-4932-9f4c-0d475d4597ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:15.447446 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.447276    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-credential-socket\") pod \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") "
Nov 03 21:04:15.447446 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.447315    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-data" pod "b8cf0dee-6ac9-4932-9f4c-0d475d4597ae" (UID: "b8cf0dee-6ac9-4932-9f4c-0d475d4597ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:15.447446 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.447326    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-workload-socket\") pod \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") "
Nov 03 21:04:15.447725 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.447427    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-kube-api-access-spdcb\" (UniqueName: \"kubernetes.io/projected/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-kube-api-access-spdcb\") pod \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") "
Nov 03 21:04:15.447725 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.447509    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-workload-socket" pod "b8cf0dee-6ac9-4932-9f4c-0d475d4597ae" (UID: "b8cf0dee-6ac9-4932-9f4c-0d475d4597ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:15.447725 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.447525    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-podinfo\") pod \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\" (UID: \"b8cf0dee-6ac9-4932-9f4c-0d475d4597ae\") "
Nov 03 21:04:15.448245 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.448156    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-credential-socket" pod "b8cf0dee-6ac9-4932-9f4c-0d475d4597ae" (UID: "b8cf0dee-6ac9-4932-9f4c-0d475d4597ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:15.448245 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.448238    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:15.448466 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.448258    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:15.448466 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.448273    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:15.448466 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.448292    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:15.450044 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.450010    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-envoy" pod "b8cf0dee-6ac9-4932-9f4c-0d475d4597ae" (UID: "b8cf0dee-6ac9-4932-9f4c-0d475d4597ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:15.450792 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.450767    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-podinfo" pod "b8cf0dee-6ac9-4932-9f4c-0d475d4597ae" (UID: "b8cf0dee-6ac9-4932-9f4c-0d475d4597ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:04:15.451016 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.450995    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-token" pod "b8cf0dee-6ac9-4932-9f4c-0d475d4597ae" (UID: "b8cf0dee-6ac9-4932-9f4c-0d475d4597ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:15.451078 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.451060    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-kube-api-access-spdcb" pod "b8cf0dee-6ac9-4932-9f4c-0d475d4597ae" (UID: "b8cf0dee-6ac9-4932-9f4c-0d475d4597ae"). InnerVolumeSpecName "kube-api-access-spdcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:15.476087 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.476064    4421 scope.go:122] "RemoveContainer" containerID="ebf0b18190f0973556f120eebc7cc6284a7be162e983760cc68a4d1232324eb6"
Nov 03 21:04:15.476394 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:04:15.476375    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebf0b18190f0973556f120eebc7cc6284a7be162e983760cc68a4d1232324eb6\": container with ID starting with ebf0b18190f0973556f120eebc7cc6284a7be162e983760cc68a4d1232324eb6 not found: ID does not exist" containerID="ebf0b18190f0973556f120eebc7cc6284a7be162e983760cc68a4d1232324eb6"
Nov 03 21:04:15.476457 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.476405    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf0b18190f0973556f120eebc7cc6284a7be162e983760cc68a4d1232324eb6"} err="failed to get container status \"ebf0b18190f0973556f120eebc7cc6284a7be162e983760cc68a4d1232324eb6\": rpc error: code = NotFound desc = could not find container \"ebf0b18190f0973556f120eebc7cc6284a7be162e983760cc68a4d1232324eb6\": container with ID starting with ebf0b18190f0973556f120eebc7cc6284a7be162e983760cc68a4d1232324eb6 not found: ID does not exist"
Nov 03 21:04:15.550494 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.550398    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:15.550494 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.550438    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spdcb\" (UniqueName: \"kubernetes.io/projected/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-kube-api-access-spdcb\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:15.550494 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.550452    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:15.550494 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.550465    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:15.550494 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.550477    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:15.771329 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.771298    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"]
Nov 03 21:04:15.775304 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:15.775276    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-c7aecb-openshift-default-c8cf956df-mfdxm"]
Nov 03 21:04:16.016163 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:16.016127    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8cf0dee-6ac9-4932-9f4c-0d475d4597ae" path="/var/lib/kubelet/pods/b8cf0dee-6ac9-4932-9f4c-0d475d4597ae/volumes"
Nov 03 21:04:19.628631 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:04:19.628571    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:04:21.972686 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:21.972651    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89"]
Nov 03 21:04:21.973173 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:21.972868    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89" podUID="4a288a11-b106-4ddc-a272-fe47967dacab" containerName="maas-api" containerID="cri-o://bc6adce0a5f952263895b0102217d948d00c2049ddd246cefffdc7231ddafde5" gracePeriod=30
Nov 03 21:04:22.300296 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:22.300269    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89"
Nov 03 21:04:22.331390 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:22.331363    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a288a11-b106-4ddc-a272-fe47967dacab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a288a11-b106-4ddc-a272-fe47967dacab-maas-api-tls\") pod \"4a288a11-b106-4ddc-a272-fe47967dacab\" (UID: \"4a288a11-b106-4ddc-a272-fe47967dacab\") "
Nov 03 21:04:22.331553 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:22.331482    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a288a11-b106-4ddc-a272-fe47967dacab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a288a11-b106-4ddc-a272-fe47967dacab-openshift-service-ca\") pod \"4a288a11-b106-4ddc-a272-fe47967dacab\" (UID: \"4a288a11-b106-4ddc-a272-fe47967dacab\") "
Nov 03 21:04:22.331553 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:22.331522    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a288a11-b106-4ddc-a272-fe47967dacab-kube-api-access-dggkr\" (UniqueName: \"kubernetes.io/projected/4a288a11-b106-4ddc-a272-fe47967dacab-kube-api-access-dggkr\") pod \"4a288a11-b106-4ddc-a272-fe47967dacab\" (UID: \"4a288a11-b106-4ddc-a272-fe47967dacab\") "
Nov 03 21:04:22.331721 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:22.331565    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a288a11-b106-4ddc-a272-fe47967dacab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a288a11-b106-4ddc-a272-fe47967dacab-metrics-certs\") pod \"4a288a11-b106-4ddc-a272-fe47967dacab\" (UID: \"4a288a11-b106-4ddc-a272-fe47967dacab\") "
Nov 03 21:04:22.331959 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:22.331927    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a288a11-b106-4ddc-a272-fe47967dacab-openshift-service-ca" pod "4a288a11-b106-4ddc-a272-fe47967dacab" (UID: "4a288a11-b106-4ddc-a272-fe47967dacab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:22.332120 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:22.332098    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a288a11-b106-4ddc-a272-fe47967dacab-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:22.334143 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:22.334117    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a288a11-b106-4ddc-a272-fe47967dacab-kube-api-access-dggkr" pod "4a288a11-b106-4ddc-a272-fe47967dacab" (UID: "4a288a11-b106-4ddc-a272-fe47967dacab"). InnerVolumeSpecName "kube-api-access-dggkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:22.334251 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:22.334159    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a288a11-b106-4ddc-a272-fe47967dacab-metrics-certs" pod "4a288a11-b106-4ddc-a272-fe47967dacab" (UID: "4a288a11-b106-4ddc-a272-fe47967dacab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:22.334328 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:22.334267    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a288a11-b106-4ddc-a272-fe47967dacab-maas-api-tls" pod "4a288a11-b106-4ddc-a272-fe47967dacab" (UID: "4a288a11-b106-4ddc-a272-fe47967dacab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:22.433827 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:22.433785    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dggkr\" (UniqueName: \"kubernetes.io/projected/4a288a11-b106-4ddc-a272-fe47967dacab-kube-api-access-dggkr\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:22.434101 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:22.434078    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a288a11-b106-4ddc-a272-fe47967dacab-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:22.434229 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:22.434216    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a288a11-b106-4ddc-a272-fe47967dacab-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:22.487687 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:22.487651    4421 generic.go:356] "Generic (PLEG): container finished" podID="4a288a11-b106-4ddc-a272-fe47967dacab" containerID="bc6adce0a5f952263895b0102217d948d00c2049ddd246cefffdc7231ddafde5" exitCode=0
Nov 03 21:04:22.487855 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:22.487696    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89"
Nov 03 21:04:22.487855 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:22.487719    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89" event={"ID":"4a288a11-b106-4ddc-a272-fe47967dacab","Type":"ContainerDied","Data":"bc6adce0a5f952263895b0102217d948d00c2049ddd246cefffdc7231ddafde5"}
Nov 03 21:04:22.487855 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:22.487770    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89" event={"ID":"4a288a11-b106-4ddc-a272-fe47967dacab","Type":"ContainerDied","Data":"1193c8de501610314574c5fc643357cfb58b6a33f6ab7c7cdf9c618cdea032c4"}
Nov 03 21:04:22.487855 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:22.487794    4421 scope.go:122] "RemoveContainer" containerID="bc6adce0a5f952263895b0102217d948d00c2049ddd246cefffdc7231ddafde5"
Nov 03 21:04:22.498587 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:22.498565    4421 scope.go:122] "RemoveContainer" containerID="bc6adce0a5f952263895b0102217d948d00c2049ddd246cefffdc7231ddafde5"
Nov 03 21:04:22.498876 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:04:22.498852    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc6adce0a5f952263895b0102217d948d00c2049ddd246cefffdc7231ddafde5\": container with ID starting with bc6adce0a5f952263895b0102217d948d00c2049ddd246cefffdc7231ddafde5 not found: ID does not exist" containerID="bc6adce0a5f952263895b0102217d948d00c2049ddd246cefffdc7231ddafde5"
Nov 03 21:04:22.498987 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:22.498881    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc6adce0a5f952263895b0102217d948d00c2049ddd246cefffdc7231ddafde5"} err="failed to get container status \"bc6adce0a5f952263895b0102217d948d00c2049ddd246cefffdc7231ddafde5\": rpc error: code = NotFound desc = could not find container \"bc6adce0a5f952263895b0102217d948d00c2049ddd246cefffdc7231ddafde5\": container with ID starting with bc6adce0a5f952263895b0102217d948d00c2049ddd246cefffdc7231ddafde5 not found: ID does not exist"
Nov 03 21:04:22.511550 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:22.511528    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89"]
Nov 03 21:04:22.515283 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:22.515265    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-db5198-c68c58f5c-45z89"]
Nov 03 21:04:24.015169 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:24.015135    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a288a11-b106-4ddc-a272-fe47967dacab" path="/var/lib/kubelet/pods/4a288a11-b106-4ddc-a272-fe47967dacab/volumes"
Nov 03 21:04:30.469434 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:30.469389    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main" containerID="cri-o://01de387c14daeb12e629246cc7a52c4ecd0b7ad67eb272e186137532d3a37fb0" gracePeriod=45
Nov 03 21:04:30.749057 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:30.749033    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 21:04:30.842400 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:30.842368    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-tmp-dir\") pod \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\" (UID: \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\") "
Nov 03 21:04:30.842588 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:30.842473    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-home\" (UniqueName: \"kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-home\") pod \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\" (UID: \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\") "
Nov 03 21:04:30.842588 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:30.842501    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-model-cache\") pod \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\" (UID: \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\") "
Nov 03 21:04:30.842588 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:30.842569    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd24563c-4cba-4ad1-9f2b-0f62b199634f-tls-certs\" (UniqueName: \"kubernetes.io/secret/bd24563c-4cba-4ad1-9f2b-0f62b199634f-tls-certs\") pod \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\" (UID: \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\") "
Nov 03 21:04:30.842818 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:30.842671    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-tmp-dir" pod "bd24563c-4cba-4ad1-9f2b-0f62b199634f" (UID: "bd24563c-4cba-4ad1-9f2b-0f62b199634f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:30.842818 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:30.842703    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-dshm\") pod \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\" (UID: \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\") "
Nov 03 21:04:30.842818 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:30.842787    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd24563c-4cba-4ad1-9f2b-0f62b199634f-kube-api-access-kbrj7\" (UniqueName: \"kubernetes.io/projected/bd24563c-4cba-4ad1-9f2b-0f62b199634f-kube-api-access-kbrj7\") pod \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\" (UID: \"bd24563c-4cba-4ad1-9f2b-0f62b199634f\") "
Nov 03 21:04:30.842962 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:30.842783    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-home" pod "bd24563c-4cba-4ad1-9f2b-0f62b199634f" (UID: "bd24563c-4cba-4ad1-9f2b-0f62b199634f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:30.842962 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:30.842809    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-model-cache" pod "bd24563c-4cba-4ad1-9f2b-0f62b199634f" (UID: "bd24563c-4cba-4ad1-9f2b-0f62b199634f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:30.843202 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:30.843183    4421 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-tmp-dir\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:30.843280 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:30.843204    4421 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-home\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:30.843280 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:30.843217    4421 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-model-cache\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:30.844994 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:30.844970    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd24563c-4cba-4ad1-9f2b-0f62b199634f-tls-certs" pod "bd24563c-4cba-4ad1-9f2b-0f62b199634f" (UID: "bd24563c-4cba-4ad1-9f2b-0f62b199634f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:30.844994 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:30.844988    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-dshm" pod "bd24563c-4cba-4ad1-9f2b-0f62b199634f" (UID: "bd24563c-4cba-4ad1-9f2b-0f62b199634f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:04:30.845157 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:30.845016    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd24563c-4cba-4ad1-9f2b-0f62b199634f-kube-api-access-kbrj7" pod "bd24563c-4cba-4ad1-9f2b-0f62b199634f" (UID: "bd24563c-4cba-4ad1-9f2b-0f62b199634f"). InnerVolumeSpecName "kube-api-access-kbrj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:30.944118 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:30.944086    4421 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd24563c-4cba-4ad1-9f2b-0f62b199634f-dshm\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:30.944118 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:30.944117    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbrj7\" (UniqueName: \"kubernetes.io/projected/bd24563c-4cba-4ad1-9f2b-0f62b199634f-kube-api-access-kbrj7\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:30.944336 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:30.944129    4421 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd24563c-4cba-4ad1-9f2b-0f62b199634f-tls-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:31.539734 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:31.539692    4421 generic.go:356] "Generic (PLEG): container finished" podID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerID="01de387c14daeb12e629246cc7a52c4ecd0b7ad67eb272e186137532d3a37fb0" exitCode=0
Nov 03 21:04:31.540229 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:31.539736    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"
Nov 03 21:04:31.540229 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:31.539780    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" event={"ID":"bd24563c-4cba-4ad1-9f2b-0f62b199634f","Type":"ContainerDied","Data":"01de387c14daeb12e629246cc7a52c4ecd0b7ad67eb272e186137532d3a37fb0"}
Nov 03 21:04:31.540229 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:31.539821    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t" event={"ID":"bd24563c-4cba-4ad1-9f2b-0f62b199634f","Type":"ContainerDied","Data":"a2200a736b42d3f20b1d2a897b971923aaeb3db6133210c64deb0c04fdf5376f"}
Nov 03 21:04:31.540229 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:31.539841    4421 scope.go:122] "RemoveContainer" containerID="01de387c14daeb12e629246cc7a52c4ecd0b7ad67eb272e186137532d3a37fb0"
Nov 03 21:04:31.551990 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:31.551959    4421 scope.go:122] "RemoveContainer" containerID="01de387c14daeb12e629246cc7a52c4ecd0b7ad67eb272e186137532d3a37fb0"
Nov 03 21:04:31.552391 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:04:31.552362    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01de387c14daeb12e629246cc7a52c4ecd0b7ad67eb272e186137532d3a37fb0\": container with ID starting with 01de387c14daeb12e629246cc7a52c4ecd0b7ad67eb272e186137532d3a37fb0 not found: ID does not exist" containerID="01de387c14daeb12e629246cc7a52c4ecd0b7ad67eb272e186137532d3a37fb0"
Nov 03 21:04:31.552524 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:31.552499    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01de387c14daeb12e629246cc7a52c4ecd0b7ad67eb272e186137532d3a37fb0"} err="failed to get container status \"01de387c14daeb12e629246cc7a52c4ecd0b7ad67eb272e186137532d3a37fb0\": rpc error: code = NotFound desc = could not find container \"01de387c14daeb12e629246cc7a52c4ecd0b7ad67eb272e186137532d3a37fb0\": container with ID starting with 01de387c14daeb12e629246cc7a52c4ecd0b7ad67eb272e186137532d3a37fb0 not found: ID does not exist"
Nov 03 21:04:31.564896 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:31.564868    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"]
Nov 03 21:04:31.569165 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:31.569144    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-085566/simulator-w2-085566-kserve-5959946fbc-56t6t"]
Nov 03 21:04:32.015704 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:32.015672    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" path="/var/lib/kubelet/pods/bd24563c-4cba-4ad1-9f2b-0f62b199634f/volumes"
Nov 03 21:04:36.206121 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:36.205996    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8"]
Nov 03 21:04:36.210241 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:36.210079    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8" podUID="b02e2fd7-7327-4777-ba31-4c683a809719" containerName="maas-api" containerID="cri-o://79fcd906f100eb3c80b8060a836220a1fb9bab4124eb725ccd789a55a62f8d46" gracePeriod=30
Nov 03 21:04:36.531025 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:36.531001    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8"
Nov 03 21:04:36.575071 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:36.575033    4421 generic.go:356] "Generic (PLEG): container finished" podID="b02e2fd7-7327-4777-ba31-4c683a809719" containerID="79fcd906f100eb3c80b8060a836220a1fb9bab4124eb725ccd789a55a62f8d46" exitCode=0
Nov 03 21:04:36.575277 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:36.575104    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8" event={"ID":"b02e2fd7-7327-4777-ba31-4c683a809719","Type":"ContainerDied","Data":"79fcd906f100eb3c80b8060a836220a1fb9bab4124eb725ccd789a55a62f8d46"}
Nov 03 21:04:36.575277 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:36.575141    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8" event={"ID":"b02e2fd7-7327-4777-ba31-4c683a809719","Type":"ContainerDied","Data":"e5a55207e65ae8d428c4e9bd68f308295f944677f3ff764854ab5dcfc6deb9ce"}
Nov 03 21:04:36.575277 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:36.575161    4421 scope.go:122] "RemoveContainer" containerID="79fcd906f100eb3c80b8060a836220a1fb9bab4124eb725ccd789a55a62f8d46"
Nov 03 21:04:36.575277 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:36.575257    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8"
Nov 03 21:04:36.578693 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:36.578666    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b02e2fd7-7327-4777-ba31-4c683a809719-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b02e2fd7-7327-4777-ba31-4c683a809719-metrics-certs\") pod \"b02e2fd7-7327-4777-ba31-4c683a809719\" (UID: \"b02e2fd7-7327-4777-ba31-4c683a809719\") "
Nov 03 21:04:36.578846 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:36.578731    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b02e2fd7-7327-4777-ba31-4c683a809719-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02e2fd7-7327-4777-ba31-4c683a809719-maas-api-tls\") pod \"b02e2fd7-7327-4777-ba31-4c683a809719\" (UID: \"b02e2fd7-7327-4777-ba31-4c683a809719\") "
Nov 03 21:04:36.581781 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:36.581741    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b02e2fd7-7327-4777-ba31-4c683a809719-maas-api-tls" pod "b02e2fd7-7327-4777-ba31-4c683a809719" (UID: "b02e2fd7-7327-4777-ba31-4c683a809719"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:36.581923 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:36.581892    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b02e2fd7-7327-4777-ba31-4c683a809719-metrics-certs" pod "b02e2fd7-7327-4777-ba31-4c683a809719" (UID: "b02e2fd7-7327-4777-ba31-4c683a809719"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:36.587959 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:36.587924    4421 scope.go:122] "RemoveContainer" containerID="79fcd906f100eb3c80b8060a836220a1fb9bab4124eb725ccd789a55a62f8d46"
Nov 03 21:04:36.588276 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:04:36.588252    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79fcd906f100eb3c80b8060a836220a1fb9bab4124eb725ccd789a55a62f8d46\": container with ID starting with 79fcd906f100eb3c80b8060a836220a1fb9bab4124eb725ccd789a55a62f8d46 not found: ID does not exist" containerID="79fcd906f100eb3c80b8060a836220a1fb9bab4124eb725ccd789a55a62f8d46"
Nov 03 21:04:36.588361 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:36.588288    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79fcd906f100eb3c80b8060a836220a1fb9bab4124eb725ccd789a55a62f8d46"} err="failed to get container status \"79fcd906f100eb3c80b8060a836220a1fb9bab4124eb725ccd789a55a62f8d46\": rpc error: code = NotFound desc = could not find container \"79fcd906f100eb3c80b8060a836220a1fb9bab4124eb725ccd789a55a62f8d46\": container with ID starting with 79fcd906f100eb3c80b8060a836220a1fb9bab4124eb725ccd789a55a62f8d46 not found: ID does not exist"
Nov 03 21:04:36.680007 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:36.679974    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b02e2fd7-7327-4777-ba31-4c683a809719-kube-api-access-fbwwp\" (UniqueName: \"kubernetes.io/projected/b02e2fd7-7327-4777-ba31-4c683a809719-kube-api-access-fbwwp\") pod \"b02e2fd7-7327-4777-ba31-4c683a809719\" (UID: \"b02e2fd7-7327-4777-ba31-4c683a809719\") "
Nov 03 21:04:36.680215 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:36.680055    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b02e2fd7-7327-4777-ba31-4c683a809719-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02e2fd7-7327-4777-ba31-4c683a809719-openshift-service-ca\") pod \"b02e2fd7-7327-4777-ba31-4c683a809719\" (UID: \"b02e2fd7-7327-4777-ba31-4c683a809719\") "
Nov 03 21:04:36.680344 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:36.680331    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b02e2fd7-7327-4777-ba31-4c683a809719-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:36.680405 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:36.680346    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02e2fd7-7327-4777-ba31-4c683a809719-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:36.680464 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:36.680409    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b02e2fd7-7327-4777-ba31-4c683a809719-openshift-service-ca" pod "b02e2fd7-7327-4777-ba31-4c683a809719" (UID: "b02e2fd7-7327-4777-ba31-4c683a809719"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:36.682408 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:36.682384    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02e2fd7-7327-4777-ba31-4c683a809719-kube-api-access-fbwwp" pod "b02e2fd7-7327-4777-ba31-4c683a809719" (UID: "b02e2fd7-7327-4777-ba31-4c683a809719"). InnerVolumeSpecName "kube-api-access-fbwwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:36.781124 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:36.781048    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbwwp\" (UniqueName: \"kubernetes.io/projected/b02e2fd7-7327-4777-ba31-4c683a809719-kube-api-access-fbwwp\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:36.781124 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:36.781075    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02e2fd7-7327-4777-ba31-4c683a809719-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:36.898038 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:36.898012    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8"]
Nov 03 21:04:36.902365 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:36.902339    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-421604-5fb5fc5cd-z25m8"]
Nov 03 21:04:38.016528 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:38.016493    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b02e2fd7-7327-4777-ba31-4c683a809719" path="/var/lib/kubelet/pods/b02e2fd7-7327-4777-ba31-4c683a809719/volumes"
Nov 03 21:04:47.119391 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:47.119349    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9"]
Nov 03 21:04:47.119845 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:47.119631    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9" podUID="afdf3167-09b7-498c-8674-31b63af5622c" containerName="maas-api" containerID="cri-o://fa633f0aedf2ecedf54908a8fe247937f538067ef00b0d7d2205fc9ceba8bc6d" gracePeriod=30
Nov 03 21:04:47.427985 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:47.427954    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9"
Nov 03 21:04:47.498088 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:47.498057    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afdf3167-09b7-498c-8674-31b63af5622c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afdf3167-09b7-498c-8674-31b63af5622c-openshift-service-ca\") pod \"afdf3167-09b7-498c-8674-31b63af5622c\" (UID: \"afdf3167-09b7-498c-8674-31b63af5622c\") "
Nov 03 21:04:47.498278 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:47.498113    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afdf3167-09b7-498c-8674-31b63af5622c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afdf3167-09b7-498c-8674-31b63af5622c-maas-api-tls\") pod \"afdf3167-09b7-498c-8674-31b63af5622c\" (UID: \"afdf3167-09b7-498c-8674-31b63af5622c\") "
Nov 03 21:04:47.498278 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:47.498168    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afdf3167-09b7-498c-8674-31b63af5622c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/afdf3167-09b7-498c-8674-31b63af5622c-metrics-certs\") pod \"afdf3167-09b7-498c-8674-31b63af5622c\" (UID: \"afdf3167-09b7-498c-8674-31b63af5622c\") "
Nov 03 21:04:47.498278 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:47.498230    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afdf3167-09b7-498c-8674-31b63af5622c-kube-api-access-2n2hk\" (UniqueName: \"kubernetes.io/projected/afdf3167-09b7-498c-8674-31b63af5622c-kube-api-access-2n2hk\") pod \"afdf3167-09b7-498c-8674-31b63af5622c\" (UID: \"afdf3167-09b7-498c-8674-31b63af5622c\") "
Nov 03 21:04:47.498507 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:47.498484    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afdf3167-09b7-498c-8674-31b63af5622c-openshift-service-ca" pod "afdf3167-09b7-498c-8674-31b63af5622c" (UID: "afdf3167-09b7-498c-8674-31b63af5622c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:47.498792 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:47.498773    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afdf3167-09b7-498c-8674-31b63af5622c-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:47.500454 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:47.500429    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afdf3167-09b7-498c-8674-31b63af5622c-maas-api-tls" pod "afdf3167-09b7-498c-8674-31b63af5622c" (UID: "afdf3167-09b7-498c-8674-31b63af5622c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:47.500542 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:47.500496    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afdf3167-09b7-498c-8674-31b63af5622c-metrics-certs" pod "afdf3167-09b7-498c-8674-31b63af5622c" (UID: "afdf3167-09b7-498c-8674-31b63af5622c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:47.500583 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:47.500532    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afdf3167-09b7-498c-8674-31b63af5622c-kube-api-access-2n2hk" pod "afdf3167-09b7-498c-8674-31b63af5622c" (UID: "afdf3167-09b7-498c-8674-31b63af5622c"). InnerVolumeSpecName "kube-api-access-2n2hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:47.600213 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:47.600183    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afdf3167-09b7-498c-8674-31b63af5622c-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:47.600213 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:47.600209    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n2hk\" (UniqueName: \"kubernetes.io/projected/afdf3167-09b7-498c-8674-31b63af5622c-kube-api-access-2n2hk\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:47.600213 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:47.600220    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afdf3167-09b7-498c-8674-31b63af5622c-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:47.637697 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:47.637664    4421 generic.go:356] "Generic (PLEG): container finished" podID="afdf3167-09b7-498c-8674-31b63af5622c" containerID="fa633f0aedf2ecedf54908a8fe247937f538067ef00b0d7d2205fc9ceba8bc6d" exitCode=0
Nov 03 21:04:47.637877 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:47.637716    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9"
Nov 03 21:04:47.637877 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:47.637746    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9" event={"ID":"afdf3167-09b7-498c-8674-31b63af5622c","Type":"ContainerDied","Data":"fa633f0aedf2ecedf54908a8fe247937f538067ef00b0d7d2205fc9ceba8bc6d"}
Nov 03 21:04:47.637877 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:47.637805    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9" event={"ID":"afdf3167-09b7-498c-8674-31b63af5622c","Type":"ContainerDied","Data":"412ebb9594ea13592ccc14835dc493463397ba0749b6bebd9ec2d8b3a5fc8b04"}
Nov 03 21:04:47.637877 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:47.637834    4421 scope.go:122] "RemoveContainer" containerID="fa633f0aedf2ecedf54908a8fe247937f538067ef00b0d7d2205fc9ceba8bc6d"
Nov 03 21:04:47.648773 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:47.648750    4421 scope.go:122] "RemoveContainer" containerID="fa633f0aedf2ecedf54908a8fe247937f538067ef00b0d7d2205fc9ceba8bc6d"
Nov 03 21:04:47.649111 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:04:47.649078    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa633f0aedf2ecedf54908a8fe247937f538067ef00b0d7d2205fc9ceba8bc6d\": container with ID starting with fa633f0aedf2ecedf54908a8fe247937f538067ef00b0d7d2205fc9ceba8bc6d not found: ID does not exist" containerID="fa633f0aedf2ecedf54908a8fe247937f538067ef00b0d7d2205fc9ceba8bc6d"
Nov 03 21:04:47.649231 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:47.649119    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa633f0aedf2ecedf54908a8fe247937f538067ef00b0d7d2205fc9ceba8bc6d"} err="failed to get container status \"fa633f0aedf2ecedf54908a8fe247937f538067ef00b0d7d2205fc9ceba8bc6d\": rpc error: code = NotFound desc = could not find container \"fa633f0aedf2ecedf54908a8fe247937f538067ef00b0d7d2205fc9ceba8bc6d\": container with ID starting with fa633f0aedf2ecedf54908a8fe247937f538067ef00b0d7d2205fc9ceba8bc6d not found: ID does not exist"
Nov 03 21:04:47.662355 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:47.662300    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9"]
Nov 03 21:04:47.665694 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:47.665673    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-085566-759f8c4797-q6sz9"]
Nov 03 21:04:48.017072 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:48.017040    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afdf3167-09b7-498c-8674-31b63af5622c" path="/var/lib/kubelet/pods/afdf3167-09b7-498c-8674-31b63af5622c/volumes"
Nov 03 21:04:49.628564 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:04:49.628481    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:04:52.651407 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:52.651366    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw"]
Nov 03 21:04:52.667337 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:52.666891    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n"]
Nov 03 21:04:55.400540 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:55.400492    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"]
Nov 03 21:04:55.401502 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:55.401403    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm" podUID="c6ba69b5-5aac-45b5-a324-1557e64456e8" containerName="istio-proxy" containerID="cri-o://5ae143c271475e2f747bff31a1a6e1cbbe93c20a5f64673e2f34dd433a0016a8" gracePeriod=30
Nov 03 21:04:55.550509 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:55.550452    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg"]
Nov 03 21:04:55.551731 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:55.551700    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg" podUID="bcc56ff3-69b2-4de1-a481-c5dc245ebe83" containerName="maas-api" containerID="cri-o://83a2baabbe9d3cf7b7cb88d704402ff4c58e6ff628599f7d7031026455bf679f" gracePeriod=30
Nov 03 21:04:55.688855 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:55.688822    4421 generic.go:356] "Generic (PLEG): container finished" podID="bcc56ff3-69b2-4de1-a481-c5dc245ebe83" containerID="83a2baabbe9d3cf7b7cb88d704402ff4c58e6ff628599f7d7031026455bf679f" exitCode=0
Nov 03 21:04:55.688974 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:55.688892    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg" event={"ID":"bcc56ff3-69b2-4de1-a481-c5dc245ebe83","Type":"ContainerDied","Data":"83a2baabbe9d3cf7b7cb88d704402ff4c58e6ff628599f7d7031026455bf679f"}
Nov 03 21:04:55.872379 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:55.872352    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg"
Nov 03 21:04:56.016618 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:56.016557    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-openshift-service-ca\") pod \"bcc56ff3-69b2-4de1-a481-c5dc245ebe83\" (UID: \"bcc56ff3-69b2-4de1-a481-c5dc245ebe83\") "
Nov 03 21:04:56.016848 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:56.016828    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-kube-api-access-lt2db\" (UniqueName: \"kubernetes.io/projected/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-kube-api-access-lt2db\") pod \"bcc56ff3-69b2-4de1-a481-c5dc245ebe83\" (UID: \"bcc56ff3-69b2-4de1-a481-c5dc245ebe83\") "
Nov 03 21:04:56.016981 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:56.016962    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-maas-api-tls\") pod \"bcc56ff3-69b2-4de1-a481-c5dc245ebe83\" (UID: \"bcc56ff3-69b2-4de1-a481-c5dc245ebe83\") "
Nov 03 21:04:56.017055 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:56.017014    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-metrics-certs\") pod \"bcc56ff3-69b2-4de1-a481-c5dc245ebe83\" (UID: \"bcc56ff3-69b2-4de1-a481-c5dc245ebe83\") "
Nov 03 21:04:56.018777 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:56.018736    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-openshift-service-ca" pod "bcc56ff3-69b2-4de1-a481-c5dc245ebe83" (UID: "bcc56ff3-69b2-4de1-a481-c5dc245ebe83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:04:56.022902 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:56.022865    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-metrics-certs" pod "bcc56ff3-69b2-4de1-a481-c5dc245ebe83" (UID: "bcc56ff3-69b2-4de1-a481-c5dc245ebe83"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:56.022902 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:56.022882    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-maas-api-tls" pod "bcc56ff3-69b2-4de1-a481-c5dc245ebe83" (UID: "bcc56ff3-69b2-4de1-a481-c5dc245ebe83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:04:56.024164 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:56.024136    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-kube-api-access-lt2db" pod "bcc56ff3-69b2-4de1-a481-c5dc245ebe83" (UID: "bcc56ff3-69b2-4de1-a481-c5dc245ebe83"). InnerVolumeSpecName "kube-api-access-lt2db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:04:56.117070 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:56.117034    4421 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:04:56.117254 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:56.117100    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm" podUID="c6ba69b5-5aac-45b5-a324-1557e64456e8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:04:56.118152 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:56.118128    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt2db\" (UniqueName: \"kubernetes.io/projected/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-kube-api-access-lt2db\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:56.118246 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:56.118155    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:56.118246 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:56.118164    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:56.118246 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:56.118173    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcc56ff3-69b2-4de1-a481-c5dc245ebe83-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:04:56.699763 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:56.699729    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg" event={"ID":"bcc56ff3-69b2-4de1-a481-c5dc245ebe83","Type":"ContainerDied","Data":"54b92a47641673fe8b53445e7cfca44aa6fac09882c495cad2832d18bde00b3d"}
Nov 03 21:04:56.700235 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:56.699799    4421 scope.go:122] "RemoveContainer" containerID="83a2baabbe9d3cf7b7cb88d704402ff4c58e6ff628599f7d7031026455bf679f"
Nov 03 21:04:56.700235 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:56.699736    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg"
Nov 03 21:04:56.734481 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:56.734443    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg"]
Nov 03 21:04:56.740353 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:56.740328    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ba89e0-544749d995-nwgwg"]
Nov 03 21:04:57.652164 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:57.652129    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw" podUID="e0672d75-59ba-490f-bc47-7b2f37ad0e1f" containerName="payload-processing" containerID="cri-o://0ef1bd0b156b4f6681d9698869e9ab4d526d8bd07290f51020d9c07252ba3b73" gracePeriod=25
Nov 03 21:04:57.668136 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:57.668107    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n" podUID="f70b4f2d-2560-4801-8137-ad1d2d0adf4b" containerName="payload-pre-processing" containerID="cri-o://b2ebf5ac7017bc5572cb9e5f318d62e5292ec93ec4b341e10ad1f8c1701dd18d" gracePeriod=25
Nov 03 21:04:58.015421 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:04:58.015390    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcc56ff3-69b2-4de1-a481-c5dc245ebe83" path="/var/lib/kubelet/pods/bcc56ff3-69b2-4de1-a481-c5dc245ebe83/volumes"
Nov 03 21:05:00.247050 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.247014    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"]
Nov 03 21:05:00.247534 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.247285    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs" podUID="d366ab06-4688-4c4b-b71b-66bb5d20167d" containerName="istio-proxy" containerID="cri-o://56e0455963275decf974ba78a3d7fa578549e25ad0546379e3ac1f0732ccb124" gracePeriod=30
Nov 03 21:05:00.665008 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.664982    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 21:05:00.725053 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.725016    4421 generic.go:356] "Generic (PLEG): container finished" podID="c6ba69b5-5aac-45b5-a324-1557e64456e8" containerID="5ae143c271475e2f747bff31a1a6e1cbbe93c20a5f64673e2f34dd433a0016a8" exitCode=0
Nov 03 21:05:00.725249 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.725107    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm" event={"ID":"c6ba69b5-5aac-45b5-a324-1557e64456e8","Type":"ContainerDied","Data":"5ae143c271475e2f747bff31a1a6e1cbbe93c20a5f64673e2f34dd433a0016a8"}
Nov 03 21:05:00.725249 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.725125    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"
Nov 03 21:05:00.725249 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.725167    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm" event={"ID":"c6ba69b5-5aac-45b5-a324-1557e64456e8","Type":"ContainerDied","Data":"e20f6378aa4c9a28382bbe14ea873750a84a8c2513d2d109c0697097494b39e8"}
Nov 03 21:05:00.725249 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.725201    4421 scope.go:122] "RemoveContainer" containerID="5ae143c271475e2f747bff31a1a6e1cbbe93c20a5f64673e2f34dd433a0016a8"
Nov 03 21:05:00.734931 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.734907    4421 scope.go:122] "RemoveContainer" containerID="5ae143c271475e2f747bff31a1a6e1cbbe93c20a5f64673e2f34dd433a0016a8"
Nov 03 21:05:00.735209 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:05:00.735187    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ae143c271475e2f747bff31a1a6e1cbbe93c20a5f64673e2f34dd433a0016a8\": container with ID starting with 5ae143c271475e2f747bff31a1a6e1cbbe93c20a5f64673e2f34dd433a0016a8 not found: ID does not exist" containerID="5ae143c271475e2f747bff31a1a6e1cbbe93c20a5f64673e2f34dd433a0016a8"
Nov 03 21:05:00.735291 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.735222    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ae143c271475e2f747bff31a1a6e1cbbe93c20a5f64673e2f34dd433a0016a8"} err="failed to get container status \"5ae143c271475e2f747bff31a1a6e1cbbe93c20a5f64673e2f34dd433a0016a8\": rpc error: code = NotFound desc = could not find container \"5ae143c271475e2f747bff31a1a6e1cbbe93c20a5f64673e2f34dd433a0016a8\": container with ID starting with 5ae143c271475e2f747bff31a1a6e1cbbe93c20a5f64673e2f34dd433a0016a8 not found: ID does not exist"
Nov 03 21:05:00.775685 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.775611    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ba69b5-5aac-45b5-a324-1557e64456e8-kube-api-access-r829h\" (UniqueName: \"kubernetes.io/projected/c6ba69b5-5aac-45b5-a324-1557e64456e8-kube-api-access-r829h\") pod \"c6ba69b5-5aac-45b5-a324-1557e64456e8\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") "
Nov 03 21:05:00.775685 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.775660    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-data\") pod \"c6ba69b5-5aac-45b5-a324-1557e64456e8\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") "
Nov 03 21:05:00.775932 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.775718    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-workload-socket\") pod \"c6ba69b5-5aac-45b5-a324-1557e64456e8\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") "
Nov 03 21:05:00.775932 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.775741    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-workload-certs\") pod \"c6ba69b5-5aac-45b5-a324-1557e64456e8\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") "
Nov 03 21:05:00.775932 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.775786    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-token\" (UniqueName: \"kubernetes.io/projected/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-token\") pod \"c6ba69b5-5aac-45b5-a324-1557e64456e8\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") "
Nov 03 21:05:00.775932 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.775874    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-envoy\") pod \"c6ba69b5-5aac-45b5-a324-1557e64456e8\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") "
Nov 03 21:05:00.776145 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.775932    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6ba69b5-5aac-45b5-a324-1557e64456e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ba69b5-5aac-45b5-a324-1557e64456e8-istiod-ca-cert\") pod \"c6ba69b5-5aac-45b5-a324-1557e64456e8\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") "
Nov 03 21:05:00.776145 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.775958    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-podinfo\") pod \"c6ba69b5-5aac-45b5-a324-1557e64456e8\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") "
Nov 03 21:05:00.776145 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.775993    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-credential-socket\") pod \"c6ba69b5-5aac-45b5-a324-1557e64456e8\" (UID: \"c6ba69b5-5aac-45b5-a324-1557e64456e8\") "
Nov 03 21:05:00.776145 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.776087    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-workload-certs" pod "c6ba69b5-5aac-45b5-a324-1557e64456e8" (UID: "c6ba69b5-5aac-45b5-a324-1557e64456e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:00.776145 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.776096    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-workload-socket" pod "c6ba69b5-5aac-45b5-a324-1557e64456e8" (UID: "c6ba69b5-5aac-45b5-a324-1557e64456e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:00.776145 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.776112    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-data" pod "c6ba69b5-5aac-45b5-a324-1557e64456e8" (UID: "c6ba69b5-5aac-45b5-a324-1557e64456e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:00.776500 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.776470    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-credential-socket" pod "c6ba69b5-5aac-45b5-a324-1557e64456e8" (UID: "c6ba69b5-5aac-45b5-a324-1557e64456e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:00.776500 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.776476    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:00.776628 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.776508    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ba69b5-5aac-45b5-a324-1557e64456e8-istiod-ca-cert" pod "c6ba69b5-5aac-45b5-a324-1557e64456e8" (UID: "c6ba69b5-5aac-45b5-a324-1557e64456e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:05:00.776628 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.776520    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:00.776628 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.776538    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:00.778234 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.778213    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-podinfo" pod "c6ba69b5-5aac-45b5-a324-1557e64456e8" (UID: "c6ba69b5-5aac-45b5-a324-1557e64456e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:05:00.778355 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.778275    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-token" pod "c6ba69b5-5aac-45b5-a324-1557e64456e8" (UID: "c6ba69b5-5aac-45b5-a324-1557e64456e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:00.778554 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.778529    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-envoy" pod "c6ba69b5-5aac-45b5-a324-1557e64456e8" (UID: "c6ba69b5-5aac-45b5-a324-1557e64456e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:00.778554 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.778535    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ba69b5-5aac-45b5-a324-1557e64456e8-kube-api-access-r829h" pod "c6ba69b5-5aac-45b5-a324-1557e64456e8" (UID: "c6ba69b5-5aac-45b5-a324-1557e64456e8"). InnerVolumeSpecName "kube-api-access-r829h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:00.878213 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.878179    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:00.878213 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.878207    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6ba69b5-5aac-45b5-a324-1557e64456e8-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:00.878213 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.878218    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:00.878452 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.878229    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6ba69b5-5aac-45b5-a324-1557e64456e8-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:00.878452 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.878238    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r829h\" (UniqueName: \"kubernetes.io/projected/c6ba69b5-5aac-45b5-a324-1557e64456e8-kube-api-access-r829h\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:00.878452 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:00.878247    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6ba69b5-5aac-45b5-a324-1557e64456e8-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:01.049459 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:01.049427    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"]
Nov 03 21:05:01.053357 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:01.053330    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-djgdm"]
Nov 03 21:05:02.015017 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:02.014987    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6ba69b5-5aac-45b5-a324-1557e64456e8" path="/var/lib/kubelet/pods/c6ba69b5-5aac-45b5-a324-1557e64456e8/volumes"
Nov 03 21:05:05.507927 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.507904    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 21:05:05.634827 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.634742    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-credential-socket\") pod \"d366ab06-4688-4c4b-b71b-66bb5d20167d\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") "
Nov 03 21:05:05.634827 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.634777    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-workload-certs\") pod \"d366ab06-4688-4c4b-b71b-66bb5d20167d\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") "
Nov 03 21:05:05.634827 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.634802    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d366ab06-4688-4c4b-b71b-66bb5d20167d-kube-api-access-65c5b\" (UniqueName: \"kubernetes.io/projected/d366ab06-4688-4c4b-b71b-66bb5d20167d-kube-api-access-65c5b\") pod \"d366ab06-4688-4c4b-b71b-66bb5d20167d\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") "
Nov 03 21:05:05.635111 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.634830    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-workload-socket\") pod \"d366ab06-4688-4c4b-b71b-66bb5d20167d\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") "
Nov 03 21:05:05.635111 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.634862    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d366ab06-4688-4c4b-b71b-66bb5d20167d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d366ab06-4688-4c4b-b71b-66bb5d20167d-istiod-ca-cert\") pod \"d366ab06-4688-4c4b-b71b-66bb5d20167d\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") "
Nov 03 21:05:05.635111 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.634892    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-token\" (UniqueName: \"kubernetes.io/projected/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-token\") pod \"d366ab06-4688-4c4b-b71b-66bb5d20167d\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") "
Nov 03 21:05:05.635111 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.634977    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-data\") pod \"d366ab06-4688-4c4b-b71b-66bb5d20167d\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") "
Nov 03 21:05:05.635111 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.635051    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-envoy\") pod \"d366ab06-4688-4c4b-b71b-66bb5d20167d\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") "
Nov 03 21:05:05.635111 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.635103    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-podinfo\") pod \"d366ab06-4688-4c4b-b71b-66bb5d20167d\" (UID: \"d366ab06-4688-4c4b-b71b-66bb5d20167d\") "
Nov 03 21:05:05.635447 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.635107    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-credential-socket" pod "d366ab06-4688-4c4b-b71b-66bb5d20167d" (UID: "d366ab06-4688-4c4b-b71b-66bb5d20167d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:05.635577 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.635207    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-workload-certs" pod "d366ab06-4688-4c4b-b71b-66bb5d20167d" (UID: "d366ab06-4688-4c4b-b71b-66bb5d20167d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:05.635715 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.635617    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:05.635945 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.635315    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-data" pod "d366ab06-4688-4c4b-b71b-66bb5d20167d" (UID: "d366ab06-4688-4c4b-b71b-66bb5d20167d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:05.635945 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.635472    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-workload-socket" pod "d366ab06-4688-4c4b-b71b-66bb5d20167d" (UID: "d366ab06-4688-4c4b-b71b-66bb5d20167d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:05.635945 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.635399    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d366ab06-4688-4c4b-b71b-66bb5d20167d-istiod-ca-cert" pod "d366ab06-4688-4c4b-b71b-66bb5d20167d" (UID: "d366ab06-4688-4c4b-b71b-66bb5d20167d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:05:05.637393 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.637366    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d366ab06-4688-4c4b-b71b-66bb5d20167d-kube-api-access-65c5b" pod "d366ab06-4688-4c4b-b71b-66bb5d20167d" (UID: "d366ab06-4688-4c4b-b71b-66bb5d20167d"). InnerVolumeSpecName "kube-api-access-65c5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:05.637837 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.637722    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-podinfo" pod "d366ab06-4688-4c4b-b71b-66bb5d20167d" (UID: "d366ab06-4688-4c4b-b71b-66bb5d20167d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:05:05.637837 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.637750    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-token" pod "d366ab06-4688-4c4b-b71b-66bb5d20167d" (UID: "d366ab06-4688-4c4b-b71b-66bb5d20167d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:05.637837 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.637798    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-envoy" pod "d366ab06-4688-4c4b-b71b-66bb5d20167d" (UID: "d366ab06-4688-4c4b-b71b-66bb5d20167d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:05.737344 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.737314    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:05.737509 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.737346    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65c5b\" (UniqueName: \"kubernetes.io/projected/d366ab06-4688-4c4b-b71b-66bb5d20167d-kube-api-access-65c5b\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:05.737509 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.737378    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:05.737509 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.737389    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d366ab06-4688-4c4b-b71b-66bb5d20167d-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:05.737509 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.737399    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:05.737509 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.737409    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:05.737509 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.737417    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:05.737509 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.737427    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d366ab06-4688-4c4b-b71b-66bb5d20167d-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:05.755670 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.755635    4421 generic.go:356] "Generic (PLEG): container finished" podID="d366ab06-4688-4c4b-b71b-66bb5d20167d" containerID="56e0455963275decf974ba78a3d7fa578549e25ad0546379e3ac1f0732ccb124" exitCode=0
Nov 03 21:05:05.755824 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.755689    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"
Nov 03 21:05:05.755824 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.755713    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs" event={"ID":"d366ab06-4688-4c4b-b71b-66bb5d20167d","Type":"ContainerDied","Data":"56e0455963275decf974ba78a3d7fa578549e25ad0546379e3ac1f0732ccb124"}
Nov 03 21:05:05.755824 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.755773    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs" event={"ID":"d366ab06-4688-4c4b-b71b-66bb5d20167d","Type":"ContainerDied","Data":"dfc29380fdaccc30d3dccef5e170b527a7843d353a876f058976ce941fa84e40"}
Nov 03 21:05:05.755824 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.755796    4421 scope.go:122] "RemoveContainer" containerID="56e0455963275decf974ba78a3d7fa578549e25ad0546379e3ac1f0732ccb124"
Nov 03 21:05:05.767332 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.767312    4421 scope.go:122] "RemoveContainer" containerID="c6b1e4c79dc0eeb303bea0d33ed7fd995f7f616229ef4c287e6e4f7c5fe371d6"
Nov 03 21:05:05.777623 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.777585    4421 scope.go:122] "RemoveContainer" containerID="56e0455963275decf974ba78a3d7fa578549e25ad0546379e3ac1f0732ccb124"
Nov 03 21:05:05.778762 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:05:05.778729    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56e0455963275decf974ba78a3d7fa578549e25ad0546379e3ac1f0732ccb124\": container with ID starting with 56e0455963275decf974ba78a3d7fa578549e25ad0546379e3ac1f0732ccb124 not found: ID does not exist" containerID="56e0455963275decf974ba78a3d7fa578549e25ad0546379e3ac1f0732ccb124"
Nov 03 21:05:05.778863 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.778776    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56e0455963275decf974ba78a3d7fa578549e25ad0546379e3ac1f0732ccb124"} err="failed to get container status \"56e0455963275decf974ba78a3d7fa578549e25ad0546379e3ac1f0732ccb124\": rpc error: code = NotFound desc = could not find container \"56e0455963275decf974ba78a3d7fa578549e25ad0546379e3ac1f0732ccb124\": container with ID starting with 56e0455963275decf974ba78a3d7fa578549e25ad0546379e3ac1f0732ccb124 not found: ID does not exist"
Nov 03 21:05:05.778863 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.778800    4421 scope.go:122] "RemoveContainer" containerID="c6b1e4c79dc0eeb303bea0d33ed7fd995f7f616229ef4c287e6e4f7c5fe371d6"
Nov 03 21:05:05.781059 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.781037    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"]
Nov 03 21:05:05.781158 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:05:05.781141    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6b1e4c79dc0eeb303bea0d33ed7fd995f7f616229ef4c287e6e4f7c5fe371d6\": container with ID starting with c6b1e4c79dc0eeb303bea0d33ed7fd995f7f616229ef4c287e6e4f7c5fe371d6 not found: ID does not exist" containerID="c6b1e4c79dc0eeb303bea0d33ed7fd995f7f616229ef4c287e6e4f7c5fe371d6"
Nov 03 21:05:05.781227 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.781164    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6b1e4c79dc0eeb303bea0d33ed7fd995f7f616229ef4c287e6e4f7c5fe371d6"} err="failed to get container status \"c6b1e4c79dc0eeb303bea0d33ed7fd995f7f616229ef4c287e6e4f7c5fe371d6\": rpc error: code = NotFound desc = could not find container \"c6b1e4c79dc0eeb303bea0d33ed7fd995f7f616229ef4c287e6e4f7c5fe371d6\": container with ID starting with c6b1e4c79dc0eeb303bea0d33ed7fd995f7f616229ef4c287e6e4f7c5fe371d6 not found: ID does not exist"
Nov 03 21:05:05.784042 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:05.784022    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-m5mhs"]
Nov 03 21:05:06.015921 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:06.015888    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d366ab06-4688-4c4b-b71b-66bb5d20167d" path="/var/lib/kubelet/pods/d366ab06-4688-4c4b-b71b-66bb5d20167d/volumes"
Nov 03 21:05:06.703306 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:06.703270    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n" podUID="f70b4f2d-2560-4801-8137-ad1d2d0adf4b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:05:06.703758 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:06.703274    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw" podUID="e0672d75-59ba-490f-bc47-7b2f37ad0e1f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:05:07.137016 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:07.136912    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7"]
Nov 03 21:05:07.152334 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:07.152289    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p"]
Nov 03 21:05:08.695425 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:08.695394    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w"]
Nov 03 21:05:08.695970 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:08.695715    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w" podUID="aa825357-78e3-4507-884b-b071ae001d58" containerName="maas-api" containerID="cri-o://3d2d8bc0ca86d634f6eabd105e5fd8fcc51440ca8c7ea98943d76f3a627ea46a" gracePeriod=30
Nov 03 21:05:08.970500 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:08.970467    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w"
Nov 03 21:05:09.070752 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:09.070712    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa825357-78e3-4507-884b-b071ae001d58-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa825357-78e3-4507-884b-b071ae001d58-openshift-service-ca\") pod \"aa825357-78e3-4507-884b-b071ae001d58\" (UID: \"aa825357-78e3-4507-884b-b071ae001d58\") "
Nov 03 21:05:09.070958 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:09.070783    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa825357-78e3-4507-884b-b071ae001d58-kube-api-access-qqvfs\" (UniqueName: \"kubernetes.io/projected/aa825357-78e3-4507-884b-b071ae001d58-kube-api-access-qqvfs\") pod \"aa825357-78e3-4507-884b-b071ae001d58\" (UID: \"aa825357-78e3-4507-884b-b071ae001d58\") "
Nov 03 21:05:09.070958 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:09.070851    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa825357-78e3-4507-884b-b071ae001d58-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa825357-78e3-4507-884b-b071ae001d58-metrics-certs\") pod \"aa825357-78e3-4507-884b-b071ae001d58\" (UID: \"aa825357-78e3-4507-884b-b071ae001d58\") "
Nov 03 21:05:09.071112 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:09.070975    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa825357-78e3-4507-884b-b071ae001d58-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa825357-78e3-4507-884b-b071ae001d58-maas-api-tls\") pod \"aa825357-78e3-4507-884b-b071ae001d58\" (UID: \"aa825357-78e3-4507-884b-b071ae001d58\") "
Nov 03 21:05:09.071112 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:09.071067    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa825357-78e3-4507-884b-b071ae001d58-openshift-service-ca" pod "aa825357-78e3-4507-884b-b071ae001d58" (UID: "aa825357-78e3-4507-884b-b071ae001d58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:05:09.071556 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:09.071460    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa825357-78e3-4507-884b-b071ae001d58-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:09.073413 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:09.073380    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa825357-78e3-4507-884b-b071ae001d58-kube-api-access-qqvfs" pod "aa825357-78e3-4507-884b-b071ae001d58" (UID: "aa825357-78e3-4507-884b-b071ae001d58"). InnerVolumeSpecName "kube-api-access-qqvfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:09.073531 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:09.073425    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa825357-78e3-4507-884b-b071ae001d58-metrics-certs" pod "aa825357-78e3-4507-884b-b071ae001d58" (UID: "aa825357-78e3-4507-884b-b071ae001d58"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:05:09.073531 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:09.073512    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa825357-78e3-4507-884b-b071ae001d58-maas-api-tls" pod "aa825357-78e3-4507-884b-b071ae001d58" (UID: "aa825357-78e3-4507-884b-b071ae001d58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:05:09.173260 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:09.173221    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqvfs\" (UniqueName: \"kubernetes.io/projected/aa825357-78e3-4507-884b-b071ae001d58-kube-api-access-qqvfs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:09.173260 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:09.173250    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa825357-78e3-4507-884b-b071ae001d58-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:09.173260 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:09.173259    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa825357-78e3-4507-884b-b071ae001d58-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:09.781102 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:09.781069    4421 generic.go:356] "Generic (PLEG): container finished" podID="aa825357-78e3-4507-884b-b071ae001d58" containerID="3d2d8bc0ca86d634f6eabd105e5fd8fcc51440ca8c7ea98943d76f3a627ea46a" exitCode=0
Nov 03 21:05:09.781586 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:09.781110    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w" event={"ID":"aa825357-78e3-4507-884b-b071ae001d58","Type":"ContainerDied","Data":"3d2d8bc0ca86d634f6eabd105e5fd8fcc51440ca8c7ea98943d76f3a627ea46a"}
Nov 03 21:05:09.781586 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:09.781119    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w"
Nov 03 21:05:09.781586 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:09.781152    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w" event={"ID":"aa825357-78e3-4507-884b-b071ae001d58","Type":"ContainerDied","Data":"5f60854a25c95d5b3748a31fe820d77011b6e8758ecf5d79550855462057a311"}
Nov 03 21:05:09.781586 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:09.781173    4421 scope.go:122] "RemoveContainer" containerID="3d2d8bc0ca86d634f6eabd105e5fd8fcc51440ca8c7ea98943d76f3a627ea46a"
Nov 03 21:05:09.791409 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:09.791387    4421 scope.go:122] "RemoveContainer" containerID="3d2d8bc0ca86d634f6eabd105e5fd8fcc51440ca8c7ea98943d76f3a627ea46a"
Nov 03 21:05:09.791727 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:05:09.791697    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d2d8bc0ca86d634f6eabd105e5fd8fcc51440ca8c7ea98943d76f3a627ea46a\": container with ID starting with 3d2d8bc0ca86d634f6eabd105e5fd8fcc51440ca8c7ea98943d76f3a627ea46a not found: ID does not exist" containerID="3d2d8bc0ca86d634f6eabd105e5fd8fcc51440ca8c7ea98943d76f3a627ea46a"
Nov 03 21:05:09.791835 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:09.791735    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d2d8bc0ca86d634f6eabd105e5fd8fcc51440ca8c7ea98943d76f3a627ea46a"} err="failed to get container status \"3d2d8bc0ca86d634f6eabd105e5fd8fcc51440ca8c7ea98943d76f3a627ea46a\": rpc error: code = NotFound desc = could not find container \"3d2d8bc0ca86d634f6eabd105e5fd8fcc51440ca8c7ea98943d76f3a627ea46a\": container with ID starting with 3d2d8bc0ca86d634f6eabd105e5fd8fcc51440ca8c7ea98943d76f3a627ea46a not found: ID does not exist"
Nov 03 21:05:09.804228 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:09.804169    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w"]
Nov 03 21:05:09.807107 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:09.807082    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-99a8fe-589b48f9f-t2j7w"]
Nov 03 21:05:10.016283 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:10.016252    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa825357-78e3-4507-884b-b071ae001d58" path="/var/lib/kubelet/pods/aa825357-78e3-4507-884b-b071ae001d58/volumes"
Nov 03 21:05:12.138337 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:12.138293    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7" podUID="a872c432-f38a-437a-904e-26bd73eaea99" containerName="payload-processing" containerID="cri-o://f72c6980c32575092b0735053e6604abe2254dc4764c935f20ab4b36769c3272" gracePeriod=25
Nov 03 21:05:12.153142 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:12.153103    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p" podUID="0f074459-f167-446e-9acf-075cdbfc4ef0" containerName="payload-pre-processing" containerID="cri-o://36dc09b47a33fab5e1e4d9eed511843c3bb9252d16c2bc25cd03892479f4ea62" gracePeriod=25
Nov 03 21:05:12.317854 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:12.317809    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7" podUID="a872c432-f38a-437a-904e-26bd73eaea99" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:05:15.610579 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:15.610542    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"]
Nov 03 21:05:15.611101 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:15.610768    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf" podUID="32697d8f-a748-4deb-8e4c-5c8a08c235f4" containerName="istio-proxy" containerID="cri-o://900d2c38acb4a4b584714b8f48c7e4b38bacec834cca965e46de926539afacf1" gracePeriod=30
Nov 03 21:05:16.703121 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:16.703079    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw" podUID="e0672d75-59ba-490f-bc47-7b2f37ad0e1f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:05:16.703537 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:16.703125    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n" podUID="f70b4f2d-2560-4801-8137-ad1d2d0adf4b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:05:17.839710 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:17.839639    4421 generic.go:356] "Generic (PLEG): container finished" podID="f70b4f2d-2560-4801-8137-ad1d2d0adf4b" containerID="b2ebf5ac7017bc5572cb9e5f318d62e5292ec93ec4b341e10ad1f8c1701dd18d" exitCode=0
Nov 03 21:05:17.840302 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:17.839778    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n" event={"ID":"f70b4f2d-2560-4801-8137-ad1d2d0adf4b","Type":"ContainerDied","Data":"b2ebf5ac7017bc5572cb9e5f318d62e5292ec93ec4b341e10ad1f8c1701dd18d"}
Nov 03 21:05:17.842343 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:17.842318    4421 generic.go:356] "Generic (PLEG): container finished" podID="e0672d75-59ba-490f-bc47-7b2f37ad0e1f" containerID="0ef1bd0b156b4f6681d9698869e9ab4d526d8bd07290f51020d9c07252ba3b73" exitCode=0
Nov 03 21:05:17.842449 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:17.842434    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw" event={"ID":"e0672d75-59ba-490f-bc47-7b2f37ad0e1f","Type":"ContainerDied","Data":"0ef1bd0b156b4f6681d9698869e9ab4d526d8bd07290f51020d9c07252ba3b73"}
Nov 03 21:05:17.961577 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:17.961552    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw"
Nov 03 21:05:17.972789 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:17.972763    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n"
Nov 03 21:05:18.067582 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:18.067553    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0672d75-59ba-490f-bc47-7b2f37ad0e1f-kube-api-access-ppbn4\" (UniqueName: \"kubernetes.io/projected/e0672d75-59ba-490f-bc47-7b2f37ad0e1f-kube-api-access-ppbn4\") pod \"e0672d75-59ba-490f-bc47-7b2f37ad0e1f\" (UID: \"e0672d75-59ba-490f-bc47-7b2f37ad0e1f\") "
Nov 03 21:05:18.067796 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:18.067660    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0672d75-59ba-490f-bc47-7b2f37ad0e1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0672d75-59ba-490f-bc47-7b2f37ad0e1f-plugins-config-volume\") pod \"e0672d75-59ba-490f-bc47-7b2f37ad0e1f\" (UID: \"e0672d75-59ba-490f-bc47-7b2f37ad0e1f\") "
Nov 03 21:05:18.067796 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:18.067711    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f70b4f2d-2560-4801-8137-ad1d2d0adf4b-kube-api-access-8ng8z\" (UniqueName: \"kubernetes.io/projected/f70b4f2d-2560-4801-8137-ad1d2d0adf4b-kube-api-access-8ng8z\") pod \"f70b4f2d-2560-4801-8137-ad1d2d0adf4b\" (UID: \"f70b4f2d-2560-4801-8137-ad1d2d0adf4b\") "
Nov 03 21:05:18.067796 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:18.067740    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f70b4f2d-2560-4801-8137-ad1d2d0adf4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f70b4f2d-2560-4801-8137-ad1d2d0adf4b-plugins-config-volume\") pod \"f70b4f2d-2560-4801-8137-ad1d2d0adf4b\" (UID: \"f70b4f2d-2560-4801-8137-ad1d2d0adf4b\") "
Nov 03 21:05:18.068148 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:18.068057    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0672d75-59ba-490f-bc47-7b2f37ad0e1f-plugins-config-volume" pod "e0672d75-59ba-490f-bc47-7b2f37ad0e1f" (UID: "e0672d75-59ba-490f-bc47-7b2f37ad0e1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:05:18.068358 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:18.068295    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0672d75-59ba-490f-bc47-7b2f37ad0e1f-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:18.068469 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:18.068446    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70b4f2d-2560-4801-8137-ad1d2d0adf4b-plugins-config-volume" pod "f70b4f2d-2560-4801-8137-ad1d2d0adf4b" (UID: "f70b4f2d-2560-4801-8137-ad1d2d0adf4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:05:18.070234 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:18.070212    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70b4f2d-2560-4801-8137-ad1d2d0adf4b-kube-api-access-8ng8z" pod "f70b4f2d-2560-4801-8137-ad1d2d0adf4b" (UID: "f70b4f2d-2560-4801-8137-ad1d2d0adf4b"). InnerVolumeSpecName "kube-api-access-8ng8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:18.070661 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:18.070640    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0672d75-59ba-490f-bc47-7b2f37ad0e1f-kube-api-access-ppbn4" pod "e0672d75-59ba-490f-bc47-7b2f37ad0e1f" (UID: "e0672d75-59ba-490f-bc47-7b2f37ad0e1f"). InnerVolumeSpecName "kube-api-access-ppbn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:18.170805 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:18.170767    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppbn4\" (UniqueName: \"kubernetes.io/projected/e0672d75-59ba-490f-bc47-7b2f37ad0e1f-kube-api-access-ppbn4\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:18.170805 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:18.170807    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ng8z\" (UniqueName: \"kubernetes.io/projected/f70b4f2d-2560-4801-8137-ad1d2d0adf4b-kube-api-access-8ng8z\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:18.171044 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:18.170822    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f70b4f2d-2560-4801-8137-ad1d2d0adf4b-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:18.848100 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:18.848069    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n"
Nov 03 21:05:18.848100 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:18.848087    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n" event={"ID":"f70b4f2d-2560-4801-8137-ad1d2d0adf4b","Type":"ContainerDied","Data":"72026110affaf6794700ce0ddc023a7d9492922094d51f92fb3855f489bc9b50"}
Nov 03 21:05:18.848677 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:18.848151    4421 scope.go:122] "RemoveContainer" containerID="b2ebf5ac7017bc5572cb9e5f318d62e5292ec93ec4b341e10ad1f8c1701dd18d"
Nov 03 21:05:18.849456 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:18.849436    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw" event={"ID":"e0672d75-59ba-490f-bc47-7b2f37ad0e1f","Type":"ContainerDied","Data":"c5418173bdf49e9c588f8ea411e0ee76edfae8fa349cf5d57ddf7262a2f2684c"}
Nov 03 21:05:18.849456 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:18.849451    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw"
Nov 03 21:05:18.858441 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:18.858421    4421 scope.go:122] "RemoveContainer" containerID="0ef1bd0b156b4f6681d9698869e9ab4d526d8bd07290f51020d9c07252ba3b73"
Nov 03 21:05:18.873609 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:18.873578    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw"]
Nov 03 21:05:18.877861 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:18.877839    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ba89e0-588b9fcb6b-wmqkw"]
Nov 03 21:05:18.889022 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:18.889000    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n"]
Nov 03 21:05:18.895620 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:18.895585    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ba89e0-67cd8fb495-sh56n"]
Nov 03 21:05:19.628121 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:05:19.628080    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:05:20.014424 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:20.014389    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0672d75-59ba-490f-bc47-7b2f37ad0e1f" path="/var/lib/kubelet/pods/e0672d75-59ba-490f-bc47-7b2f37ad0e1f/volumes"
Nov 03 21:05:20.014868 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:20.014801    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f70b4f2d-2560-4801-8137-ad1d2d0adf4b" path="/var/lib/kubelet/pods/f70b4f2d-2560-4801-8137-ad1d2d0adf4b/volumes"
Nov 03 21:05:20.877379 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:20.877292    4421 generic.go:356] "Generic (PLEG): container finished" podID="32697d8f-a748-4deb-8e4c-5c8a08c235f4" containerID="900d2c38acb4a4b584714b8f48c7e4b38bacec834cca965e46de926539afacf1" exitCode=0
Nov 03 21:05:20.877531 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:20.877367    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf" event={"ID":"32697d8f-a748-4deb-8e4c-5c8a08c235f4","Type":"ContainerDied","Data":"900d2c38acb4a4b584714b8f48c7e4b38bacec834cca965e46de926539afacf1"}
Nov 03 21:05:20.877531 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:20.877471    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf" event={"ID":"32697d8f-a748-4deb-8e4c-5c8a08c235f4","Type":"ContainerDied","Data":"76c7d3b37e3660dd067d5ec737cc9b837bd85ba31988ab0b5521648a6ff45bec"}
Nov 03 21:05:20.877531 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:20.877506    4421 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76c7d3b37e3660dd067d5ec737cc9b837bd85ba31988ab0b5521648a6ff45bec"
Nov 03 21:05:20.890012 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:20.889985    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 21:05:21.002374 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.002342    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-podinfo\") pod \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") "
Nov 03 21:05:21.002581 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.002398    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istiod-ca-cert\") pod \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") "
Nov 03 21:05:21.002581 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.002439    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-token\" (UniqueName: \"kubernetes.io/projected/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-token\") pod \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") "
Nov 03 21:05:21.002581 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.002470    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-workload-socket\") pod \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") "
Nov 03 21:05:21.002581 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.002528    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-envoy\") pod \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") "
Nov 03 21:05:21.002855 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.002632    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-credential-socket\") pod \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") "
Nov 03 21:05:21.002855 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.002660    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-data\") pod \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") "
Nov 03 21:05:21.002855 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.002676    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-workload-certs\") pod \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") "
Nov 03 21:05:21.002855 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.002712    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32697d8f-a748-4deb-8e4c-5c8a08c235f4-kube-api-access-kxpbs\" (UniqueName: \"kubernetes.io/projected/32697d8f-a748-4deb-8e4c-5c8a08c235f4-kube-api-access-kxpbs\") pod \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\" (UID: \"32697d8f-a748-4deb-8e4c-5c8a08c235f4\") "
Nov 03 21:05:21.003068 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.002938    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-workload-socket" pod "32697d8f-a748-4deb-8e4c-5c8a08c235f4" (UID: "32697d8f-a748-4deb-8e4c-5c8a08c235f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:21.003068 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.002992    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-credential-socket" pod "32697d8f-a748-4deb-8e4c-5c8a08c235f4" (UID: "32697d8f-a748-4deb-8e4c-5c8a08c235f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:21.003068 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.003008    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istiod-ca-cert" pod "32697d8f-a748-4deb-8e4c-5c8a08c235f4" (UID: "32697d8f-a748-4deb-8e4c-5c8a08c235f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:05:21.003404 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.003109    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:21.003404 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.003129    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:21.003404 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.003143    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:21.003404 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.003142    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-workload-certs" pod "32697d8f-a748-4deb-8e4c-5c8a08c235f4" (UID: "32697d8f-a748-4deb-8e4c-5c8a08c235f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:21.003404 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.003157    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-data" pod "32697d8f-a748-4deb-8e4c-5c8a08c235f4" (UID: "32697d8f-a748-4deb-8e4c-5c8a08c235f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:21.004911 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.004876    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-podinfo" pod "32697d8f-a748-4deb-8e4c-5c8a08c235f4" (UID: "32697d8f-a748-4deb-8e4c-5c8a08c235f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:05:21.004911 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.004890    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-token" pod "32697d8f-a748-4deb-8e4c-5c8a08c235f4" (UID: "32697d8f-a748-4deb-8e4c-5c8a08c235f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:21.005069 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.005006    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32697d8f-a748-4deb-8e4c-5c8a08c235f4-kube-api-access-kxpbs" pod "32697d8f-a748-4deb-8e4c-5c8a08c235f4" (UID: "32697d8f-a748-4deb-8e4c-5c8a08c235f4"). InnerVolumeSpecName "kube-api-access-kxpbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:21.005303 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.005290    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-envoy" pod "32697d8f-a748-4deb-8e4c-5c8a08c235f4" (UID: "32697d8f-a748-4deb-8e4c-5c8a08c235f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:21.078320 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.078275    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"]
Nov 03 21:05:21.083034 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.083007    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"]
Nov 03 21:05:21.094442 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.094403    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"]
Nov 03 21:05:21.102307 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.102283    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"]
Nov 03 21:05:21.104851 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.104805    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:21.104851 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.104847    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:21.105069 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.104862    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:21.105069 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.104877    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32697d8f-a748-4deb-8e4c-5c8a08c235f4-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:21.105069 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.104892    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxpbs\" (UniqueName: \"kubernetes.io/projected/32697d8f-a748-4deb-8e4c-5c8a08c235f4-kube-api-access-kxpbs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:21.105069 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.104907    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32697d8f-a748-4deb-8e4c-5c8a08c235f4-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:21.310456 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.310412    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p" podUID="0f074459-f167-446e-9acf-075cdbfc4ef0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:05:21.882057 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.882021    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"
Nov 03 21:05:21.906635 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.906611    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"]
Nov 03 21:05:21.912091 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:21.912065    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-99a8fe-openshift-default-579f8df677-vx5bf"]
Nov 03 21:05:22.015433 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:22.015402    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32697d8f-a748-4deb-8e4c-5c8a08c235f4" path="/var/lib/kubelet/pods/32697d8f-a748-4deb-8e4c-5c8a08c235f4/volumes"
Nov 03 21:05:22.317475 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:22.317433    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7" podUID="a872c432-f38a-437a-904e-26bd73eaea99" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:05:27.858531 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:27.858494    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn"]
Nov 03 21:05:31.310165 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:31.310129    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p" podUID="0f074459-f167-446e-9acf-075cdbfc4ef0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:05:32.316957 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:05:32.316929    4421 logging.go:55] [core] [Channel #18963 SubChannel #18964]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.178:50052", ServerName: "10.134.0.178:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.178:50052: connect: connection refused"
Nov 03 21:05:32.410139 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.410068    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7"
Nov 03 21:05:32.459293 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.459266    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p"
Nov 03 21:05:32.535026 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.534996    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f074459-f167-446e-9acf-075cdbfc4ef0-kube-api-access-lp66t\" (UniqueName: \"kubernetes.io/projected/0f074459-f167-446e-9acf-075cdbfc4ef0-kube-api-access-lp66t\") pod \"0f074459-f167-446e-9acf-075cdbfc4ef0\" (UID: \"0f074459-f167-446e-9acf-075cdbfc4ef0\") "
Nov 03 21:05:32.535220 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.535052    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a872c432-f38a-437a-904e-26bd73eaea99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a872c432-f38a-437a-904e-26bd73eaea99-plugins-config-volume\") pod \"a872c432-f38a-437a-904e-26bd73eaea99\" (UID: \"a872c432-f38a-437a-904e-26bd73eaea99\") "
Nov 03 21:05:32.535220 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.535091    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f074459-f167-446e-9acf-075cdbfc4ef0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f074459-f167-446e-9acf-075cdbfc4ef0-plugins-config-volume\") pod \"0f074459-f167-446e-9acf-075cdbfc4ef0\" (UID: \"0f074459-f167-446e-9acf-075cdbfc4ef0\") "
Nov 03 21:05:32.535220 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.535117    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a872c432-f38a-437a-904e-26bd73eaea99-kube-api-access-9smpw\" (UniqueName: \"kubernetes.io/projected/a872c432-f38a-437a-904e-26bd73eaea99-kube-api-access-9smpw\") pod \"a872c432-f38a-437a-904e-26bd73eaea99\" (UID: \"a872c432-f38a-437a-904e-26bd73eaea99\") "
Nov 03 21:05:32.535546 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.535518    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a872c432-f38a-437a-904e-26bd73eaea99-plugins-config-volume" pod "a872c432-f38a-437a-904e-26bd73eaea99" (UID: "a872c432-f38a-437a-904e-26bd73eaea99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:05:32.535546 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.535530    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f074459-f167-446e-9acf-075cdbfc4ef0-plugins-config-volume" pod "0f074459-f167-446e-9acf-075cdbfc4ef0" (UID: "0f074459-f167-446e-9acf-075cdbfc4ef0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:05:32.537406 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.537384    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f074459-f167-446e-9acf-075cdbfc4ef0-kube-api-access-lp66t" pod "0f074459-f167-446e-9acf-075cdbfc4ef0" (UID: "0f074459-f167-446e-9acf-075cdbfc4ef0"). InnerVolumeSpecName "kube-api-access-lp66t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:32.537485 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.537418    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a872c432-f38a-437a-904e-26bd73eaea99-kube-api-access-9smpw" pod "a872c432-f38a-437a-904e-26bd73eaea99" (UID: "a872c432-f38a-437a-904e-26bd73eaea99"). InnerVolumeSpecName "kube-api-access-9smpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:32.636710 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.636680    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp66t\" (UniqueName: \"kubernetes.io/projected/0f074459-f167-446e-9acf-075cdbfc4ef0-kube-api-access-lp66t\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:32.636710 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.636705    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a872c432-f38a-437a-904e-26bd73eaea99-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:32.636710 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.636714    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f074459-f167-446e-9acf-075cdbfc4ef0-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:32.636924 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.636724    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9smpw\" (UniqueName: \"kubernetes.io/projected/a872c432-f38a-437a-904e-26bd73eaea99-kube-api-access-9smpw\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:32.859766 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.859724    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn" podUID="3f2eb82c-c1c9-4895-97bb-0ca733b765f3" containerName="payload-processing" containerID="cri-o://544c28eeb8b30ffe14c7a6b83852a70e70ac213fa239300a9dcd498734a8bbfb" gracePeriod=25
Nov 03 21:05:32.937098 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.937065    4421 generic.go:356] "Generic (PLEG): container finished" podID="0f074459-f167-446e-9acf-075cdbfc4ef0" containerID="36dc09b47a33fab5e1e4d9eed511843c3bb9252d16c2bc25cd03892479f4ea62" exitCode=0
Nov 03 21:05:32.937221 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.937112    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p"
Nov 03 21:05:32.937221 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.937155    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p" event={"ID":"0f074459-f167-446e-9acf-075cdbfc4ef0","Type":"ContainerDied","Data":"36dc09b47a33fab5e1e4d9eed511843c3bb9252d16c2bc25cd03892479f4ea62"}
Nov 03 21:05:32.937332 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.937227    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p" event={"ID":"0f074459-f167-446e-9acf-075cdbfc4ef0","Type":"ContainerDied","Data":"5e66415c1e207c935a9a84cafb57c8ccd10cf723213a5a13f69ff53a4461a081"}
Nov 03 21:05:32.937332 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.937260    4421 scope.go:122] "RemoveContainer" containerID="36dc09b47a33fab5e1e4d9eed511843c3bb9252d16c2bc25cd03892479f4ea62"
Nov 03 21:05:32.938631 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.938589    4421 generic.go:356] "Generic (PLEG): container finished" podID="a872c432-f38a-437a-904e-26bd73eaea99" containerID="f72c6980c32575092b0735053e6604abe2254dc4764c935f20ab4b36769c3272" exitCode=0
Nov 03 21:05:32.938700 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.938665    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7"
Nov 03 21:05:32.938700 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.938679    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7" event={"ID":"a872c432-f38a-437a-904e-26bd73eaea99","Type":"ContainerDied","Data":"f72c6980c32575092b0735053e6604abe2254dc4764c935f20ab4b36769c3272"}
Nov 03 21:05:32.938789 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.938731    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7" event={"ID":"a872c432-f38a-437a-904e-26bd73eaea99","Type":"ContainerDied","Data":"194a1e8676c7a1e4f8fc87b61eb25a71a8700f857fbd9ab3be89fd70cb090d33"}
Nov 03 21:05:32.964763 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.964740    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p"]
Nov 03 21:05:32.970354 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.970334    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-99a8fe-7c69f5ccf-cqg2p"]
Nov 03 21:05:32.972575 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.972554    4421 scope.go:122] "RemoveContainer" containerID="36dc09b47a33fab5e1e4d9eed511843c3bb9252d16c2bc25cd03892479f4ea62"
Nov 03 21:05:32.972931 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:05:32.972907    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36dc09b47a33fab5e1e4d9eed511843c3bb9252d16c2bc25cd03892479f4ea62\": container with ID starting with 36dc09b47a33fab5e1e4d9eed511843c3bb9252d16c2bc25cd03892479f4ea62 not found: ID does not exist" containerID="36dc09b47a33fab5e1e4d9eed511843c3bb9252d16c2bc25cd03892479f4ea62"
Nov 03 21:05:32.973006 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.972944    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36dc09b47a33fab5e1e4d9eed511843c3bb9252d16c2bc25cd03892479f4ea62"} err="failed to get container status \"36dc09b47a33fab5e1e4d9eed511843c3bb9252d16c2bc25cd03892479f4ea62\": rpc error: code = NotFound desc = could not find container \"36dc09b47a33fab5e1e4d9eed511843c3bb9252d16c2bc25cd03892479f4ea62\": container with ID starting with 36dc09b47a33fab5e1e4d9eed511843c3bb9252d16c2bc25cd03892479f4ea62 not found: ID does not exist"
Nov 03 21:05:32.973006 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.972967    4421 scope.go:122] "RemoveContainer" containerID="f72c6980c32575092b0735053e6604abe2254dc4764c935f20ab4b36769c3272"
Nov 03 21:05:32.981851 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.981835    4421 scope.go:122] "RemoveContainer" containerID="f72c6980c32575092b0735053e6604abe2254dc4764c935f20ab4b36769c3272"
Nov 03 21:05:32.982159 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:05:32.982138    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f72c6980c32575092b0735053e6604abe2254dc4764c935f20ab4b36769c3272\": container with ID starting with f72c6980c32575092b0735053e6604abe2254dc4764c935f20ab4b36769c3272 not found: ID does not exist" containerID="f72c6980c32575092b0735053e6604abe2254dc4764c935f20ab4b36769c3272"
Nov 03 21:05:32.982236 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.982169    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f72c6980c32575092b0735053e6604abe2254dc4764c935f20ab4b36769c3272"} err="failed to get container status \"f72c6980c32575092b0735053e6604abe2254dc4764c935f20ab4b36769c3272\": rpc error: code = NotFound desc = could not find container \"f72c6980c32575092b0735053e6604abe2254dc4764c935f20ab4b36769c3272\": container with ID starting with f72c6980c32575092b0735053e6604abe2254dc4764c935f20ab4b36769c3272 not found: ID does not exist"
Nov 03 21:05:32.984077 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.984057    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7"]
Nov 03 21:05:32.985321 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:32.985303    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7"]
Nov 03 21:05:33.104676 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:33.104652    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn"
Nov 03 21:05:33.241342 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:33.241308    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f2eb82c-c1c9-4895-97bb-0ca733b765f3-kube-api-access-l2mzc\" (UniqueName: \"kubernetes.io/projected/3f2eb82c-c1c9-4895-97bb-0ca733b765f3-kube-api-access-l2mzc\") pod \"3f2eb82c-c1c9-4895-97bb-0ca733b765f3\" (UID: \"3f2eb82c-c1c9-4895-97bb-0ca733b765f3\") "
Nov 03 21:05:33.241485 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:33.241389    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f2eb82c-c1c9-4895-97bb-0ca733b765f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f2eb82c-c1c9-4895-97bb-0ca733b765f3-plugins-config-volume\") pod \"3f2eb82c-c1c9-4895-97bb-0ca733b765f3\" (UID: \"3f2eb82c-c1c9-4895-97bb-0ca733b765f3\") "
Nov 03 21:05:33.241788 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:33.241765    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f2eb82c-c1c9-4895-97bb-0ca733b765f3-plugins-config-volume" pod "3f2eb82c-c1c9-4895-97bb-0ca733b765f3" (UID: "3f2eb82c-c1c9-4895-97bb-0ca733b765f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:05:33.243562 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:33.243546    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2eb82c-c1c9-4895-97bb-0ca733b765f3-kube-api-access-l2mzc" pod "3f2eb82c-c1c9-4895-97bb-0ca733b765f3" (UID: "3f2eb82c-c1c9-4895-97bb-0ca733b765f3"). InnerVolumeSpecName "kube-api-access-l2mzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:33.317400 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:33.317361    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-99a8fe-69d4c86766-drnt7" podUID="a872c432-f38a-437a-904e-26bd73eaea99" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.178:50052\" within 1s: context deadline exceeded"
Nov 03 21:05:33.343112 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:33.343079    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f2eb82c-c1c9-4895-97bb-0ca733b765f3-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:33.343112 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:33.343109    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2mzc\" (UniqueName: \"kubernetes.io/projected/3f2eb82c-c1c9-4895-97bb-0ca733b765f3-kube-api-access-l2mzc\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:33.946852 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:33.946821    4421 generic.go:356] "Generic (PLEG): container finished" podID="3f2eb82c-c1c9-4895-97bb-0ca733b765f3" containerID="544c28eeb8b30ffe14c7a6b83852a70e70ac213fa239300a9dcd498734a8bbfb" exitCode=0
Nov 03 21:05:33.947071 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:33.946870    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn" event={"ID":"3f2eb82c-c1c9-4895-97bb-0ca733b765f3","Type":"ContainerDied","Data":"544c28eeb8b30ffe14c7a6b83852a70e70ac213fa239300a9dcd498734a8bbfb"}
Nov 03 21:05:33.947071 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:33.946881    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn"
Nov 03 21:05:33.947071 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:33.946922    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn" event={"ID":"3f2eb82c-c1c9-4895-97bb-0ca733b765f3","Type":"ContainerDied","Data":"9663ec291b15e7b6a09a29777502d42e4e47eee93e9187887cade15371cf6fea"}
Nov 03 21:05:33.947071 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:33.946952    4421 scope.go:122] "RemoveContainer" containerID="544c28eeb8b30ffe14c7a6b83852a70e70ac213fa239300a9dcd498734a8bbfb"
Nov 03 21:05:33.956725 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:33.956708    4421 scope.go:122] "RemoveContainer" containerID="544c28eeb8b30ffe14c7a6b83852a70e70ac213fa239300a9dcd498734a8bbfb"
Nov 03 21:05:33.956985 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:05:33.956966    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"544c28eeb8b30ffe14c7a6b83852a70e70ac213fa239300a9dcd498734a8bbfb\": container with ID starting with 544c28eeb8b30ffe14c7a6b83852a70e70ac213fa239300a9dcd498734a8bbfb not found: ID does not exist" containerID="544c28eeb8b30ffe14c7a6b83852a70e70ac213fa239300a9dcd498734a8bbfb"
Nov 03 21:05:33.957040 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:33.956994    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"544c28eeb8b30ffe14c7a6b83852a70e70ac213fa239300a9dcd498734a8bbfb"} err="failed to get container status \"544c28eeb8b30ffe14c7a6b83852a70e70ac213fa239300a9dcd498734a8bbfb\": rpc error: code = NotFound desc = could not find container \"544c28eeb8b30ffe14c7a6b83852a70e70ac213fa239300a9dcd498734a8bbfb\": container with ID starting with 544c28eeb8b30ffe14c7a6b83852a70e70ac213fa239300a9dcd498734a8bbfb not found: ID does not exist"
Nov 03 21:05:33.970715 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:33.970692    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn"]
Nov 03 21:05:33.978091 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:33.978072    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f3631f-5bc8d4db88-nmljn"]
Nov 03 21:05:34.014260 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:34.014231    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f074459-f167-446e-9acf-075cdbfc4ef0" path="/var/lib/kubelet/pods/0f074459-f167-446e-9acf-075cdbfc4ef0/volumes"
Nov 03 21:05:34.014626 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:34.014613    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f2eb82c-c1c9-4895-97bb-0ca733b765f3" path="/var/lib/kubelet/pods/3f2eb82c-c1c9-4895-97bb-0ca733b765f3/volumes"
Nov 03 21:05:34.014942 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:34.014930    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a872c432-f38a-437a-904e-26bd73eaea99" path="/var/lib/kubelet/pods/a872c432-f38a-437a-904e-26bd73eaea99/volumes"
Nov 03 21:05:36.101830 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.101788    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main" containerID="cri-o://91504c45ebd819efa3f91cfc8a06dddec47324d63a55f46a862bdcb126d74f0c" gracePeriod=45
Nov 03 21:05:36.106336 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.106303    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" containerID="cri-o://04ed2b4ac68785febd2e5842a09e13766a49d4f3f6895fcccd82dc98d2bada6d" gracePeriod=45
Nov 03 21:05:36.115893 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.115861    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" containerID="cri-o://e8d7b451765fe5297901715a171e45acaa202a41f4d9a03a603b8ec3a87ef3a8" gracePeriod=45
Nov 03 21:05:36.120534 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.120493    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": EOF"
Nov 03 21:05:36.122462 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.122434    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main" containerID="cri-o://8753e602801d6ae0203d12dea13adf632b561a41059e7807a0c29b27d5ed8041" gracePeriod=45
Nov 03 21:05:36.282969 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.282924    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": dial tcp 10.134.0.68:8000: connect: connection refused"
Nov 03 21:05:36.386568 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.386537    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 21:05:36.423449 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.423423    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 21:05:36.426528 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.426502    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 21:05:36.439462 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.439439    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 21:05:36.475658 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.475625    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-model-cache\") pod \"5b6f055a-4376-4113-aaaa-335ffbd9091e\" (UID: \"5b6f055a-4376-4113-aaaa-335ffbd9091e\") "
Nov 03 21:05:36.475865 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.475680    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-dshm\") pod \"5b6f055a-4376-4113-aaaa-335ffbd9091e\" (UID: \"5b6f055a-4376-4113-aaaa-335ffbd9091e\") "
Nov 03 21:05:36.475865 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.475715    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-home\" (UniqueName: \"kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-home\") pod \"5b6f055a-4376-4113-aaaa-335ffbd9091e\" (UID: \"5b6f055a-4376-4113-aaaa-335ffbd9091e\") "
Nov 03 21:05:36.475990 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.475862    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-tmp-dir\") pod \"5b6f055a-4376-4113-aaaa-335ffbd9091e\" (UID: \"5b6f055a-4376-4113-aaaa-335ffbd9091e\") "
Nov 03 21:05:36.475990 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.475923    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b6f055a-4376-4113-aaaa-335ffbd9091e-kube-api-access-4csdz\" (UniqueName: \"kubernetes.io/projected/5b6f055a-4376-4113-aaaa-335ffbd9091e-kube-api-access-4csdz\") pod \"5b6f055a-4376-4113-aaaa-335ffbd9091e\" (UID: \"5b6f055a-4376-4113-aaaa-335ffbd9091e\") "
Nov 03 21:05:36.475990 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.475975    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b6f055a-4376-4113-aaaa-335ffbd9091e-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b6f055a-4376-4113-aaaa-335ffbd9091e-tls-certs\") pod \"5b6f055a-4376-4113-aaaa-335ffbd9091e\" (UID: \"5b6f055a-4376-4113-aaaa-335ffbd9091e\") "
Nov 03 21:05:36.476151 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.476075    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-home" pod "5b6f055a-4376-4113-aaaa-335ffbd9091e" (UID: "5b6f055a-4376-4113-aaaa-335ffbd9091e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:36.476151 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.476114    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-model-cache" pod "5b6f055a-4376-4113-aaaa-335ffbd9091e" (UID: "5b6f055a-4376-4113-aaaa-335ffbd9091e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:36.476314 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.476284    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-tmp-dir" pod "5b6f055a-4376-4113-aaaa-335ffbd9091e" (UID: "5b6f055a-4376-4113-aaaa-335ffbd9091e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:36.476462 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.476446    4421 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-model-cache\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:36.476541 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.476468    4421 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-home\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:36.476541 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.476482    4421 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-tmp-dir\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:36.478503 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.478474    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b6f055a-4376-4113-aaaa-335ffbd9091e-tls-certs" pod "5b6f055a-4376-4113-aaaa-335ffbd9091e" (UID: "5b6f055a-4376-4113-aaaa-335ffbd9091e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:05:36.478646 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.478550    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b6f055a-4376-4113-aaaa-335ffbd9091e-kube-api-access-4csdz" pod "5b6f055a-4376-4113-aaaa-335ffbd9091e" (UID: "5b6f055a-4376-4113-aaaa-335ffbd9091e"). InnerVolumeSpecName "kube-api-access-4csdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:36.478877 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.478855    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-dshm" pod "5b6f055a-4376-4113-aaaa-335ffbd9091e" (UID: "5b6f055a-4376-4113-aaaa-335ffbd9091e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:36.577266 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.577236    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-tls-certs\") pod \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\" (UID: \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\") "
Nov 03 21:05:36.577470 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.577305    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4abd92cb-973c-4d9d-9a7e-003109eef888-kube-api-access-p9t4x\" (UniqueName: \"kubernetes.io/projected/4abd92cb-973c-4d9d-9a7e-003109eef888-kube-api-access-p9t4x\") pod \"4abd92cb-973c-4d9d-9a7e-003109eef888\" (UID: \"4abd92cb-973c-4d9d-9a7e-003109eef888\") "
Nov 03 21:05:36.577470 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.577331    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-tmp-dir\") pod \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\" (UID: \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\") "
Nov 03 21:05:36.577470 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.577351    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-home\" (UniqueName: \"kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-home\") pod \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\" (UID: \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\") "
Nov 03 21:05:36.577470 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.577381    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/188223ef-2d2c-4895-9f16-98b1b45aaf6f-kube-api-access-pjkd9\" (UniqueName: \"kubernetes.io/projected/188223ef-2d2c-4895-9f16-98b1b45aaf6f-kube-api-access-pjkd9\") pod \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\" (UID: \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\") "
Nov 03 21:05:36.577470 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.577426    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-dshm\") pod \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\" (UID: \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\") "
Nov 03 21:05:36.577470 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.577456    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-home\" (UniqueName: \"kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-home\") pod \"4abd92cb-973c-4d9d-9a7e-003109eef888\" (UID: \"4abd92cb-973c-4d9d-9a7e-003109eef888\") "
Nov 03 21:05:36.577863 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.577484    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-tmp-dir\") pod \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\" (UID: \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\") "
Nov 03 21:05:36.577863 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.577524    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-dshm\") pod \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\" (UID: \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\") "
Nov 03 21:05:36.577863 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.577577    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4abd92cb-973c-4d9d-9a7e-003109eef888-tls-certs\" (UniqueName: \"kubernetes.io/secret/4abd92cb-973c-4d9d-9a7e-003109eef888-tls-certs\") pod \"4abd92cb-973c-4d9d-9a7e-003109eef888\" (UID: \"4abd92cb-973c-4d9d-9a7e-003109eef888\") "
Nov 03 21:05:36.577863 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.577633    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-model-cache\") pod \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\" (UID: \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\") "
Nov 03 21:05:36.577863 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.577685    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-kube-api-access-kpdw5\" (UniqueName: \"kubernetes.io/projected/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-kube-api-access-kpdw5\") pod \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\" (UID: \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\") "
Nov 03 21:05:36.577863 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.577706    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-tmp-dir" pod "ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" (UID: "ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:36.577863 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.577750    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-model-cache\") pod \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\" (UID: \"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08\") "
Nov 03 21:05:36.577863 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.577802    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-home\" (UniqueName: \"kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-home\") pod \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\" (UID: \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\") "
Nov 03 21:05:36.578440 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.577901    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-model-cache\") pod \"4abd92cb-973c-4d9d-9a7e-003109eef888\" (UID: \"4abd92cb-973c-4d9d-9a7e-003109eef888\") "
Nov 03 21:05:36.578440 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.577902    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-home" pod "4abd92cb-973c-4d9d-9a7e-003109eef888" (UID: "4abd92cb-973c-4d9d-9a7e-003109eef888"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:36.578440 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.577924    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-tmp-dir" pod "188223ef-2d2c-4895-9f16-98b1b45aaf6f" (UID: "188223ef-2d2c-4895-9f16-98b1b45aaf6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:36.578440 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.577958    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/188223ef-2d2c-4895-9f16-98b1b45aaf6f-tls-certs\" (UniqueName: \"kubernetes.io/secret/188223ef-2d2c-4895-9f16-98b1b45aaf6f-tls-certs\") pod \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\" (UID: \"188223ef-2d2c-4895-9f16-98b1b45aaf6f\") "
Nov 03 21:05:36.578440 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.577994    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-tmp-dir\") pod \"4abd92cb-973c-4d9d-9a7e-003109eef888\" (UID: \"4abd92cb-973c-4d9d-9a7e-003109eef888\") "
Nov 03 21:05:36.578440 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.578043    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-dshm\") pod \"4abd92cb-973c-4d9d-9a7e-003109eef888\" (UID: \"4abd92cb-973c-4d9d-9a7e-003109eef888\") "
Nov 03 21:05:36.578440 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.577958    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-home" pod "ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" (UID: "ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:36.578440 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.578392    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-model-cache" pod "188223ef-2d2c-4895-9f16-98b1b45aaf6f" (UID: "188223ef-2d2c-4895-9f16-98b1b45aaf6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:36.578440 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.578410    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-home" pod "188223ef-2d2c-4895-9f16-98b1b45aaf6f" (UID: "188223ef-2d2c-4895-9f16-98b1b45aaf6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:36.578440 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.578425    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-model-cache" pod "ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" (UID: "ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:36.579017 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.578537    4421 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-model-cache\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:36.579017 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.578592    4421 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-home\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:36.579017 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.578635    4421 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b6f055a-4376-4113-aaaa-335ffbd9091e-tls-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:36.579017 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.578634    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-tmp-dir" pod "4abd92cb-973c-4d9d-9a7e-003109eef888" (UID: "4abd92cb-973c-4d9d-9a7e-003109eef888"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:36.579017 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.578649    4421 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b6f055a-4376-4113-aaaa-335ffbd9091e-dshm\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:36.579017 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.578664    4421 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-tmp-dir\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:36.579017 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.578676    4421 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-home\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:36.579017 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.578688    4421 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-home\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:36.579017 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.578701    4421 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-tmp-dir\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:36.579017 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.578713    4421 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-model-cache\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:36.579017 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.578726    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4csdz\" (UniqueName: \"kubernetes.io/projected/5b6f055a-4376-4113-aaaa-335ffbd9091e-kube-api-access-4csdz\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:36.579017 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.578815    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-model-cache" pod "4abd92cb-973c-4d9d-9a7e-003109eef888" (UID: "4abd92cb-973c-4d9d-9a7e-003109eef888"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:36.580376 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.580346    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4abd92cb-973c-4d9d-9a7e-003109eef888-kube-api-access-p9t4x" pod "4abd92cb-973c-4d9d-9a7e-003109eef888" (UID: "4abd92cb-973c-4d9d-9a7e-003109eef888"). InnerVolumeSpecName "kube-api-access-p9t4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:36.580988 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.580963    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-dshm" pod "4abd92cb-973c-4d9d-9a7e-003109eef888" (UID: "4abd92cb-973c-4d9d-9a7e-003109eef888"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:36.580988 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.580976    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188223ef-2d2c-4895-9f16-98b1b45aaf6f-kube-api-access-pjkd9" pod "188223ef-2d2c-4895-9f16-98b1b45aaf6f" (UID: "188223ef-2d2c-4895-9f16-98b1b45aaf6f"). InnerVolumeSpecName "kube-api-access-pjkd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:36.581222 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.581204    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-dshm" pod "188223ef-2d2c-4895-9f16-98b1b45aaf6f" (UID: "188223ef-2d2c-4895-9f16-98b1b45aaf6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:36.581342 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.581318    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-tls-certs" pod "ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" (UID: "ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:05:36.581467 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.581346    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-kube-api-access-kpdw5" pod "ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" (UID: "ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08"). InnerVolumeSpecName "kube-api-access-kpdw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:05:36.581467 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.581429    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4abd92cb-973c-4d9d-9a7e-003109eef888-tls-certs" pod "4abd92cb-973c-4d9d-9a7e-003109eef888" (UID: "4abd92cb-973c-4d9d-9a7e-003109eef888"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:05:36.581679 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.581660    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-dshm" pod "ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" (UID: "ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:05:36.582195 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.582177    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/188223ef-2d2c-4895-9f16-98b1b45aaf6f-tls-certs" pod "188223ef-2d2c-4895-9f16-98b1b45aaf6f" (UID: "188223ef-2d2c-4895-9f16-98b1b45aaf6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:05:36.679755 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.679671    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpdw5\" (UniqueName: \"kubernetes.io/projected/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-kube-api-access-kpdw5\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:36.679755 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.679700    4421 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-model-cache\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:36.679755 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.679710    4421 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188223ef-2d2c-4895-9f16-98b1b45aaf6f-tls-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:36.679755 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.679719    4421 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-tmp-dir\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:36.679755 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.679726    4421 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4abd92cb-973c-4d9d-9a7e-003109eef888-dshm\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:36.679755 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.679735    4421 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-tls-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:36.679755 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.679742    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9t4x\" (UniqueName: \"kubernetes.io/projected/4abd92cb-973c-4d9d-9a7e-003109eef888-kube-api-access-p9t4x\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:36.679755 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.679751    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjkd9\" (UniqueName: \"kubernetes.io/projected/188223ef-2d2c-4895-9f16-98b1b45aaf6f-kube-api-access-pjkd9\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:36.679755 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.679759    4421 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08-dshm\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:36.680168 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.679767    4421 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188223ef-2d2c-4895-9f16-98b1b45aaf6f-dshm\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:36.680168 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.679775    4421 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4abd92cb-973c-4d9d-9a7e-003109eef888-tls-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:05:36.971920 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.971881    4421 generic.go:356] "Generic (PLEG): container finished" podID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerID="e8d7b451765fe5297901715a171e45acaa202a41f4d9a03a603b8ec3a87ef3a8" exitCode=0
Nov 03 21:05:36.972125 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.971946    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"
Nov 03 21:05:36.972125 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.971971    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" event={"ID":"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08","Type":"ContainerDied","Data":"e8d7b451765fe5297901715a171e45acaa202a41f4d9a03a603b8ec3a87ef3a8"}
Nov 03 21:05:36.972125 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.972017    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf" event={"ID":"ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08","Type":"ContainerDied","Data":"535beb0d922cb590f39bd0bf7319140b0c83a0a9e557437021c03694787e3b85"}
Nov 03 21:05:36.972125 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.972035    4421 scope.go:122] "RemoveContainer" containerID="e8d7b451765fe5297901715a171e45acaa202a41f4d9a03a603b8ec3a87ef3a8"
Nov 03 21:05:36.973700 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.973672    4421 generic.go:356] "Generic (PLEG): container finished" podID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerID="91504c45ebd819efa3f91cfc8a06dddec47324d63a55f46a862bdcb126d74f0c" exitCode=0
Nov 03 21:05:36.973846 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.973737    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"
Nov 03 21:05:36.973846 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.973763    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" event={"ID":"188223ef-2d2c-4895-9f16-98b1b45aaf6f","Type":"ContainerDied","Data":"91504c45ebd819efa3f91cfc8a06dddec47324d63a55f46a862bdcb126d74f0c"}
Nov 03 21:05:36.973846 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.973800    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg" event={"ID":"188223ef-2d2c-4895-9f16-98b1b45aaf6f","Type":"ContainerDied","Data":"0adddaf540953be7ec4a143b6a405bea39fd0da48cb6742a7aaefb0eda938f83"}
Nov 03 21:05:36.975284 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.975255    4421 generic.go:356] "Generic (PLEG): container finished" podID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerID="04ed2b4ac68785febd2e5842a09e13766a49d4f3f6895fcccd82dc98d2bada6d" exitCode=0
Nov 03 21:05:36.975459 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.975282    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" event={"ID":"5b6f055a-4376-4113-aaaa-335ffbd9091e","Type":"ContainerDied","Data":"04ed2b4ac68785febd2e5842a09e13766a49d4f3f6895fcccd82dc98d2bada6d"}
Nov 03 21:05:36.975459 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.975310    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl" event={"ID":"5b6f055a-4376-4113-aaaa-335ffbd9091e","Type":"ContainerDied","Data":"26815f5311d63a0d2ebb79b44799dab2869ef947939ad9b5af9f3e9366d72d7b"}
Nov 03 21:05:36.975459 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.975310    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"
Nov 03 21:05:36.977200 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.977170    4421 generic.go:356] "Generic (PLEG): container finished" podID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerID="8753e602801d6ae0203d12dea13adf632b561a41059e7807a0c29b27d5ed8041" exitCode=0
Nov 03 21:05:36.977309 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.977231    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"
Nov 03 21:05:36.977309 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.977239    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" event={"ID":"4abd92cb-973c-4d9d-9a7e-003109eef888","Type":"ContainerDied","Data":"8753e602801d6ae0203d12dea13adf632b561a41059e7807a0c29b27d5ed8041"}
Nov 03 21:05:36.977309 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.977264    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn" event={"ID":"4abd92cb-973c-4d9d-9a7e-003109eef888","Type":"ContainerDied","Data":"7b5d3d5073761450aced7cd42952a1e1993f8d346f88d2cee8e7b4eaf3e1f955"}
Nov 03 21:05:36.984180 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.984160    4421 scope.go:122] "RemoveContainer" containerID="e8d7b451765fe5297901715a171e45acaa202a41f4d9a03a603b8ec3a87ef3a8"
Nov 03 21:05:36.984490 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:05:36.984468    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d7b451765fe5297901715a171e45acaa202a41f4d9a03a603b8ec3a87ef3a8\": container with ID starting with e8d7b451765fe5297901715a171e45acaa202a41f4d9a03a603b8ec3a87ef3a8 not found: ID does not exist" containerID="e8d7b451765fe5297901715a171e45acaa202a41f4d9a03a603b8ec3a87ef3a8"
Nov 03 21:05:36.984565 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.984498    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d7b451765fe5297901715a171e45acaa202a41f4d9a03a603b8ec3a87ef3a8"} err="failed to get container status \"e8d7b451765fe5297901715a171e45acaa202a41f4d9a03a603b8ec3a87ef3a8\": rpc error: code = NotFound desc = could not find container \"e8d7b451765fe5297901715a171e45acaa202a41f4d9a03a603b8ec3a87ef3a8\": container with ID starting with e8d7b451765fe5297901715a171e45acaa202a41f4d9a03a603b8ec3a87ef3a8 not found: ID does not exist"
Nov 03 21:05:36.984565 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.984519    4421 scope.go:122] "RemoveContainer" containerID="91504c45ebd819efa3f91cfc8a06dddec47324d63a55f46a862bdcb126d74f0c"
Nov 03 21:05:36.994461 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.994439    4421 scope.go:122] "RemoveContainer" containerID="91504c45ebd819efa3f91cfc8a06dddec47324d63a55f46a862bdcb126d74f0c"
Nov 03 21:05:36.995219 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:05:36.995200    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91504c45ebd819efa3f91cfc8a06dddec47324d63a55f46a862bdcb126d74f0c\": container with ID starting with 91504c45ebd819efa3f91cfc8a06dddec47324d63a55f46a862bdcb126d74f0c not found: ID does not exist" containerID="91504c45ebd819efa3f91cfc8a06dddec47324d63a55f46a862bdcb126d74f0c"
Nov 03 21:05:36.995319 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.995223    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91504c45ebd819efa3f91cfc8a06dddec47324d63a55f46a862bdcb126d74f0c"} err="failed to get container status \"91504c45ebd819efa3f91cfc8a06dddec47324d63a55f46a862bdcb126d74f0c\": rpc error: code = NotFound desc = could not find container \"91504c45ebd819efa3f91cfc8a06dddec47324d63a55f46a862bdcb126d74f0c\": container with ID starting with 91504c45ebd819efa3f91cfc8a06dddec47324d63a55f46a862bdcb126d74f0c not found: ID does not exist"
Nov 03 21:05:36.995319 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.995239    4421 scope.go:122] "RemoveContainer" containerID="04ed2b4ac68785febd2e5842a09e13766a49d4f3f6895fcccd82dc98d2bada6d"
Nov 03 21:05:36.996764 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:36.996747    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"]
Nov 03 21:05:37.000256 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:37.000237    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f3631f/premium-w0-f3631f-kserve-6749dbcb5f-hsfpf"]
Nov 03 21:05:37.004025 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:37.004009    4421 scope.go:122] "RemoveContainer" containerID="04ed2b4ac68785febd2e5842a09e13766a49d4f3f6895fcccd82dc98d2bada6d"
Nov 03 21:05:37.004292 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:05:37.004274    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04ed2b4ac68785febd2e5842a09e13766a49d4f3f6895fcccd82dc98d2bada6d\": container with ID starting with 04ed2b4ac68785febd2e5842a09e13766a49d4f3f6895fcccd82dc98d2bada6d not found: ID does not exist" containerID="04ed2b4ac68785febd2e5842a09e13766a49d4f3f6895fcccd82dc98d2bada6d"
Nov 03 21:05:37.004335 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:37.004298    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04ed2b4ac68785febd2e5842a09e13766a49d4f3f6895fcccd82dc98d2bada6d"} err="failed to get container status \"04ed2b4ac68785febd2e5842a09e13766a49d4f3f6895fcccd82dc98d2bada6d\": rpc error: code = NotFound desc = could not find container \"04ed2b4ac68785febd2e5842a09e13766a49d4f3f6895fcccd82dc98d2bada6d\": container with ID starting with 04ed2b4ac68785febd2e5842a09e13766a49d4f3f6895fcccd82dc98d2bada6d not found: ID does not exist"
Nov 03 21:05:37.004335 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:37.004314    4421 scope.go:122] "RemoveContainer" containerID="8753e602801d6ae0203d12dea13adf632b561a41059e7807a0c29b27d5ed8041"
Nov 03 21:05:37.012308 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:37.012292    4421 scope.go:122] "RemoveContainer" containerID="8753e602801d6ae0203d12dea13adf632b561a41059e7807a0c29b27d5ed8041"
Nov 03 21:05:37.012550 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:05:37.012531    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8753e602801d6ae0203d12dea13adf632b561a41059e7807a0c29b27d5ed8041\": container with ID starting with 8753e602801d6ae0203d12dea13adf632b561a41059e7807a0c29b27d5ed8041 not found: ID does not exist" containerID="8753e602801d6ae0203d12dea13adf632b561a41059e7807a0c29b27d5ed8041"
Nov 03 21:05:37.012661 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:37.012556    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8753e602801d6ae0203d12dea13adf632b561a41059e7807a0c29b27d5ed8041"} err="failed to get container status \"8753e602801d6ae0203d12dea13adf632b561a41059e7807a0c29b27d5ed8041\": rpc error: code = NotFound desc = could not find container \"8753e602801d6ae0203d12dea13adf632b561a41059e7807a0c29b27d5ed8041\": container with ID starting with 8753e602801d6ae0203d12dea13adf632b561a41059e7807a0c29b27d5ed8041 not found: ID does not exist"
Nov 03 21:05:37.015467 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:37.015448    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"]
Nov 03 21:05:37.019421 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:37.019404    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f3631f/simulator-w0-f3631f-kserve-b4fb8c546-6f2qn"]
Nov 03 21:05:37.030154 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:37.030131    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"]
Nov 03 21:05:37.034513 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:37.034493    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f3631f/distinct-w0-f3631f-kserve-7f9ff489c4-67fpl"]
Nov 03 21:05:37.045704 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:37.045686    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"]
Nov 03 21:05:37.052430 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:37.052406    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f3631f/distinct-2-w0-f3631f-kserve-6d4d6ffcbf-ntzwg"]
Nov 03 21:05:38.013909 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:38.013882    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" path="/var/lib/kubelet/pods/188223ef-2d2c-4895-9f16-98b1b45aaf6f/volumes"
Nov 03 21:05:38.014322 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:38.014304    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" path="/var/lib/kubelet/pods/4abd92cb-973c-4d9d-9a7e-003109eef888/volumes"
Nov 03 21:05:38.014681 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:38.014667    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" path="/var/lib/kubelet/pods/5b6f055a-4376-4113-aaaa-335ffbd9091e/volumes"
Nov 03 21:05:38.015302 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:05:38.015289    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" path="/var/lib/kubelet/pods/ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08/volumes"
Nov 03 21:05:49.627444 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:05:49.627408    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:06:00.957060 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:00.957022    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"]
Nov 03 21:06:00.959751 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:00.957298    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv" podUID="e103d59c-e7b4-4713-a826-d3a8834f947a" containerName="istio-proxy" containerID="cri-o://aedf83c4e6a2ef5dbdb3cd08339a54da45bd4bd0fa18a46028ee37acfec4b82b" gracePeriod=30
Nov 03 21:06:01.953626 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:01.953566    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cfb485dff-mjxfv_6f1f72d4-3cc6-4ae2-b9ac-a399a5b9a34c/authorino/0.log"
Nov 03 21:06:02.560157 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:02.560132    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jb45k_7e5c45c5-1f48-4b84-b7ae-244cd6e80b9f/kuadrant-console-plugin/0.log"
Nov 03 21:06:04.530393 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:04.530360    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-2725f_1c4bb3bf-caf2-4932-8b65-00189e8d41fb/manager/0.log"
Nov 03 21:06:05.444492 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:05.444456    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-687b69bd9-6n6dr_2bd8fabe-6f14-458d-ae20-bc4b71333adf/manager/0.log"
Nov 03 21:06:05.609076 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:05.609043    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-8lzxb_c23a53b9-ca79-461a-a8f6-20618ffc35aa/server/0.log"
Nov 03 21:06:05.791255 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:05.791183    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-99ntv_b347c3ff-a128-4fd2-8c07-dfea5c60a431/manager/0.log"
Nov 03 21:06:06.006971 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.006947    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-4bbvd_1dcbd7f2-5145-444c-9b8f-7035eb14b64e/proxy/0.log"
Nov 03 21:06:06.119670 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.119638    4421 generic.go:356] "Generic (PLEG): container finished" podID="e103d59c-e7b4-4713-a826-d3a8834f947a" containerID="aedf83c4e6a2ef5dbdb3cd08339a54da45bd4bd0fa18a46028ee37acfec4b82b" exitCode=0
Nov 03 21:06:06.119829 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.119745    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv" event={"ID":"e103d59c-e7b4-4713-a826-d3a8834f947a","Type":"ContainerDied","Data":"aedf83c4e6a2ef5dbdb3cd08339a54da45bd4bd0fa18a46028ee37acfec4b82b"}
Nov 03 21:06:06.202252 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.202226    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 21:06:06.259661 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.259616    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-workload-certs\") pod \"e103d59c-e7b4-4713-a826-d3a8834f947a\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") "
Nov 03 21:06:06.259843 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.259675    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-data\") pod \"e103d59c-e7b4-4713-a826-d3a8834f947a\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") "
Nov 03 21:06:06.259843 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.259732    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e103d59c-e7b4-4713-a826-d3a8834f947a-kube-api-access-zvb9t\" (UniqueName: \"kubernetes.io/projected/e103d59c-e7b4-4713-a826-d3a8834f947a-kube-api-access-zvb9t\") pod \"e103d59c-e7b4-4713-a826-d3a8834f947a\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") "
Nov 03 21:06:06.259843 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.259767    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-workload-socket\") pod \"e103d59c-e7b4-4713-a826-d3a8834f947a\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") "
Nov 03 21:06:06.259843 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.259804    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-credential-socket\") pod \"e103d59c-e7b4-4713-a826-d3a8834f947a\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") "
Nov 03 21:06:06.259843 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.259831    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-token\" (UniqueName: \"kubernetes.io/projected/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-token\") pod \"e103d59c-e7b4-4713-a826-d3a8834f947a\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") "
Nov 03 21:06:06.260150 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.259873    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-envoy\") pod \"e103d59c-e7b4-4713-a826-d3a8834f947a\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") "
Nov 03 21:06:06.260150 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.259915    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e103d59c-e7b4-4713-a826-d3a8834f947a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e103d59c-e7b4-4713-a826-d3a8834f947a-istiod-ca-cert\") pod \"e103d59c-e7b4-4713-a826-d3a8834f947a\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") "
Nov 03 21:06:06.260150 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.259916    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-workload-certs" pod "e103d59c-e7b4-4713-a826-d3a8834f947a" (UID: "e103d59c-e7b4-4713-a826-d3a8834f947a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:06:06.260150 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.259974    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-data" pod "e103d59c-e7b4-4713-a826-d3a8834f947a" (UID: "e103d59c-e7b4-4713-a826-d3a8834f947a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:06:06.260150 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.260028    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-podinfo\") pod \"e103d59c-e7b4-4713-a826-d3a8834f947a\" (UID: \"e103d59c-e7b4-4713-a826-d3a8834f947a\") "
Nov 03 21:06:06.260404 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.260167    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-credential-socket" pod "e103d59c-e7b4-4713-a826-d3a8834f947a" (UID: "e103d59c-e7b4-4713-a826-d3a8834f947a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:06:06.260404 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.260279    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-workload-socket" pod "e103d59c-e7b4-4713-a826-d3a8834f947a" (UID: "e103d59c-e7b4-4713-a826-d3a8834f947a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:06:06.260404 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.260329    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e103d59c-e7b4-4713-a826-d3a8834f947a-istiod-ca-cert" pod "e103d59c-e7b4-4713-a826-d3a8834f947a" (UID: "e103d59c-e7b4-4713-a826-d3a8834f947a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:06:06.260531 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.260417    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:06.260531 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.260436    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:06.260531 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.260452    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:06.260531 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.260465    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e103d59c-e7b4-4713-a826-d3a8834f947a-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:06.260531 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.260479    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:06.262439 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.262417    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-podinfo" pod "e103d59c-e7b4-4713-a826-d3a8834f947a" (UID: "e103d59c-e7b4-4713-a826-d3a8834f947a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:06:06.262439 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.262426    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-envoy" pod "e103d59c-e7b4-4713-a826-d3a8834f947a" (UID: "e103d59c-e7b4-4713-a826-d3a8834f947a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:06:06.262439 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.262431    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e103d59c-e7b4-4713-a826-d3a8834f947a-kube-api-access-zvb9t" pod "e103d59c-e7b4-4713-a826-d3a8834f947a" (UID: "e103d59c-e7b4-4713-a826-d3a8834f947a"). InnerVolumeSpecName "kube-api-access-zvb9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:06:06.262756 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.262734    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-token" pod "e103d59c-e7b4-4713-a826-d3a8834f947a" (UID: "e103d59c-e7b4-4713-a826-d3a8834f947a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:06:06.361568 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.361492    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvb9t\" (UniqueName: \"kubernetes.io/projected/e103d59c-e7b4-4713-a826-d3a8834f947a-kube-api-access-zvb9t\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:06.361568 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.361518    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:06.361568 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.361528    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:06.361568 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:06.361535    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e103d59c-e7b4-4713-a826-d3a8834f947a-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:07.125681 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:07.125643    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv" event={"ID":"e103d59c-e7b4-4713-a826-d3a8834f947a","Type":"ContainerDied","Data":"64f003fb24bb2c58be60a837f4b8ca01dd16c5a9488eb7202561ab6be3f41fe6"}
Nov 03 21:06:07.125681 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:07.125674    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"
Nov 03 21:06:07.126277 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:07.125704    4421 scope.go:122] "RemoveContainer" containerID="aedf83c4e6a2ef5dbdb3cd08339a54da45bd4bd0fa18a46028ee37acfec4b82b"
Nov 03 21:06:07.148762 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:07.148734    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"]
Nov 03 21:06:07.153037 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:07.153012    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-5bfwv"]
Nov 03 21:06:08.014701 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:08.014661    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e103d59c-e7b4-4713-a826-d3a8834f947a" path="/var/lib/kubelet/pods/e103d59c-e7b4-4713-a826-d3a8834f947a/volumes"
Nov 03 21:06:10.748824 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.748789    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"]
Nov 03 21:06:10.749390 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749303    4421 cpu_manager.go:414] "containerMap: removing container" podUID="0a2fda22-a9be-4bf5-8b2b-0e994221c92b" containerName="istio-proxy"
Nov 03 21:06:10.749390 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749316    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a2fda22-a9be-4bf5-8b2b-0e994221c92b" containerName="istio-proxy"
Nov 03 21:06:10.749390 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749337    4421 cpu_manager.go:414] "containerMap: removing container" podUID="657622a4-6298-4496-b2c3-edce95190d04" containerName="istio-proxy"
Nov 03 21:06:10.749390 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749345    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="657622a4-6298-4496-b2c3-edce95190d04" containerName="istio-proxy"
Nov 03 21:06:10.749390 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749355    4421 cpu_manager.go:414] "containerMap: removing container" podUID="bcc56ff3-69b2-4de1-a481-c5dc245ebe83" containerName="maas-api"
Nov 03 21:06:10.749390 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749363    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcc56ff3-69b2-4de1-a481-c5dc245ebe83" containerName="maas-api"
Nov 03 21:06:10.749390 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749374    4421 cpu_manager.go:414] "containerMap: removing container" podUID="5b495a05-8e6f-4b43-b9d3-20ad6b499731" containerName="payload-processing"
Nov 03 21:06:10.749390 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749383    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b495a05-8e6f-4b43-b9d3-20ad6b499731" containerName="payload-processing"
Nov 03 21:06:10.749390 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749396    4421 cpu_manager.go:414] "containerMap: removing container" podUID="56233474-fdc7-4bcb-aa2a-db87ac074168" containerName="istio-proxy"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749404    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56233474-fdc7-4bcb-aa2a-db87ac074168" containerName="istio-proxy"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749416    4421 cpu_manager.go:414] "containerMap: removing container" podUID="d366ab06-4688-4c4b-b71b-66bb5d20167d" containerName="istio-proxy"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749422    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d366ab06-4688-4c4b-b71b-66bb5d20167d" containerName="istio-proxy"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749433    4421 cpu_manager.go:414] "containerMap: removing container" podUID="c9adc7d2-3c5b-42e1-b790-20db31c15593" containerName="istio-proxy"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749441    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9adc7d2-3c5b-42e1-b790-20db31c15593" containerName="istio-proxy"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749450    4421 cpu_manager.go:414] "containerMap: removing container" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749457    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749465    4421 cpu_manager.go:414] "containerMap: removing container" podUID="4a288a11-b106-4ddc-a272-fe47967dacab" containerName="maas-api"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749472    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a288a11-b106-4ddc-a272-fe47967dacab" containerName="maas-api"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749485    4421 cpu_manager.go:414] "containerMap: removing container" podUID="a872c432-f38a-437a-904e-26bd73eaea99" containerName="payload-processing"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749492    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a872c432-f38a-437a-904e-26bd73eaea99" containerName="payload-processing"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749502    4421 cpu_manager.go:414] "containerMap: removing container" podUID="b02e2fd7-7327-4777-ba31-4c683a809719" containerName="maas-api"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749508    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02e2fd7-7327-4777-ba31-4c683a809719" containerName="maas-api"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749517    4421 cpu_manager.go:414] "containerMap: removing container" podUID="6eb20c2e-30b6-4bdd-9b91-fc996180e96d" containerName="payload-pre-processing"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749523    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eb20c2e-30b6-4bdd-9b91-fc996180e96d" containerName="payload-pre-processing"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749535    4421 cpu_manager.go:414] "containerMap: removing container" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749542    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749552    4421 cpu_manager.go:414] "containerMap: removing container" podUID="0f074459-f167-446e-9acf-075cdbfc4ef0" containerName="payload-pre-processing"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749559    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f074459-f167-446e-9acf-075cdbfc4ef0" containerName="payload-pre-processing"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749569    4421 cpu_manager.go:414] "containerMap: removing container" podUID="bcd47c56-cc43-41ff-951b-82c5768ad799" containerName="payload-pre-processing"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749575    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcd47c56-cc43-41ff-951b-82c5768ad799" containerName="payload-pre-processing"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749590    4421 cpu_manager.go:414] "containerMap: removing container" podUID="aa825357-78e3-4507-884b-b071ae001d58" containerName="maas-api"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749627    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa825357-78e3-4507-884b-b071ae001d58" containerName="maas-api"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749637    4421 cpu_manager.go:414] "containerMap: removing container" podUID="c6e7fd23-0f9d-4ec7-9a57-1326c2a63287" containerName="maas-api"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749643    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6e7fd23-0f9d-4ec7-9a57-1326c2a63287" containerName="maas-api"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749654    4421 cpu_manager.go:414] "containerMap: removing container" podUID="ad241f3c-90c0-4f0f-b90c-47eec0af0245" containerName="istio-proxy"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749660    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad241f3c-90c0-4f0f-b90c-47eec0af0245" containerName="istio-proxy"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749672    4421 cpu_manager.go:414] "containerMap: removing container" podUID="afdf3167-09b7-498c-8674-31b63af5622c" containerName="maas-api"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749678    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afdf3167-09b7-498c-8674-31b63af5622c" containerName="maas-api"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749688    4421 cpu_manager.go:414] "containerMap: removing container" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749695    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749705    4421 cpu_manager.go:414] "containerMap: removing container" podUID="e0672d75-59ba-490f-bc47-7b2f37ad0e1f" containerName="payload-processing"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749712    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0672d75-59ba-490f-bc47-7b2f37ad0e1f" containerName="payload-processing"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749723    4421 cpu_manager.go:414] "containerMap: removing container" podUID="a4b81cef-71a5-4458-97c9-f617fc049a8b" containerName="maas-api"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749729    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4b81cef-71a5-4458-97c9-f617fc049a8b" containerName="maas-api"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749739    4421 cpu_manager.go:414] "containerMap: removing container" podUID="c6ba69b5-5aac-45b5-a324-1557e64456e8" containerName="istio-proxy"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749746    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6ba69b5-5aac-45b5-a324-1557e64456e8" containerName="istio-proxy"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749755    4421 cpu_manager.go:414] "containerMap: removing container" podUID="feb6a504-7873-440d-a013-36654e702307" containerName="payload-processing"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749762    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feb6a504-7873-440d-a013-36654e702307" containerName="payload-processing"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749772    4421 cpu_manager.go:414] "containerMap: removing container" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749778    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749789    4421 cpu_manager.go:414] "containerMap: removing container" podUID="56233474-fdc7-4bcb-aa2a-db87ac074168" containerName="istio-proxy"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749796    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56233474-fdc7-4bcb-aa2a-db87ac074168" containerName="istio-proxy"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749809    4421 cpu_manager.go:414] "containerMap: removing container" podUID="68b472a9-c812-4308-895f-fd5193af34f6" containerName="cleanup"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749815    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68b472a9-c812-4308-895f-fd5193af34f6" containerName="cleanup"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749831    4421 cpu_manager.go:414] "containerMap: removing container" podUID="6db98514-abcc-481f-b53b-e313c205155a" containerName="istio-proxy"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749838    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db98514-abcc-481f-b53b-e313c205155a" containerName="istio-proxy"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749848    4421 cpu_manager.go:414] "containerMap: removing container" podUID="b8cf0dee-6ac9-4932-9f4c-0d475d4597ae" containerName="istio-proxy"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749854    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8cf0dee-6ac9-4932-9f4c-0d475d4597ae" containerName="istio-proxy"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749867    4421 cpu_manager.go:414] "containerMap: removing container" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749874    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749884    4421 cpu_manager.go:414] "containerMap: removing container" podUID="3f2eb82c-c1c9-4895-97bb-0ca733b765f3" containerName="payload-processing"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749891    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f2eb82c-c1c9-4895-97bb-0ca733b765f3" containerName="payload-processing"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749901    4421 cpu_manager.go:414] "containerMap: removing container" podUID="7caf12c0-4e78-4105-bc0c-7a7e8937e3ac" containerName="cleanup"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749908    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7caf12c0-4e78-4105-bc0c-7a7e8937e3ac" containerName="cleanup"
Nov 03 21:06:10.749894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749918    4421 cpu_manager.go:414] "containerMap: removing container" podUID="e103d59c-e7b4-4713-a826-d3a8834f947a" containerName="istio-proxy"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749925    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e103d59c-e7b4-4713-a826-d3a8834f947a" containerName="istio-proxy"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749933    4421 cpu_manager.go:414] "containerMap: removing container" podUID="32697d8f-a748-4deb-8e4c-5c8a08c235f4" containerName="istio-proxy"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749940    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32697d8f-a748-4deb-8e4c-5c8a08c235f4" containerName="istio-proxy"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749951    4421 cpu_manager.go:414] "containerMap: removing container" podUID="f70b4f2d-2560-4801-8137-ad1d2d0adf4b" containerName="payload-pre-processing"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.749957    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f70b4f2d-2560-4801-8137-ad1d2d0adf4b" containerName="payload-pre-processing"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750033    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6db98514-abcc-481f-b53b-e313c205155a" containerName="istio-proxy"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750045    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8cf0dee-6ac9-4932-9f4c-0d475d4597ae" containerName="istio-proxy"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750053    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b6f055a-4376-4113-aaaa-335ffbd9091e" containerName="main"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750060    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="629e965e-1f05-4fdd-a1ef-40aa9da9e3a0" containerName="cleanup"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750068    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f2eb82c-c1c9-4895-97bb-0ca733b765f3" containerName="payload-processing"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750077    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e103d59c-e7b4-4713-a826-d3a8834f947a" containerName="istio-proxy"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750087    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="32697d8f-a748-4deb-8e4c-5c8a08c235f4" containerName="istio-proxy"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750094    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f70b4f2d-2560-4801-8137-ad1d2d0adf4b" containerName="payload-pre-processing"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750103    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="657622a4-6298-4496-b2c3-edce95190d04" containerName="istio-proxy"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750112    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcc56ff3-69b2-4de1-a481-c5dc245ebe83" containerName="maas-api"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750120    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b495a05-8e6f-4b43-b9d3-20ad6b499731" containerName="payload-processing"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750127    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="56233474-fdc7-4bcb-aa2a-db87ac074168" containerName="istio-proxy"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750134    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d366ab06-4688-4c4b-b71b-66bb5d20167d" containerName="istio-proxy"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750141    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9adc7d2-3c5b-42e1-b790-20db31c15593" containerName="istio-proxy"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750148    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea1d6b64-3d8a-4b5a-b37f-c2b110cddb08" containerName="main"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750156    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a288a11-b106-4ddc-a272-fe47967dacab" containerName="maas-api"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750164    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a872c432-f38a-437a-904e-26bd73eaea99" containerName="payload-processing"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750172    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="b02e2fd7-7327-4777-ba31-4c683a809719" containerName="maas-api"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750179    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eb20c2e-30b6-4bdd-9b91-fc996180e96d" containerName="payload-pre-processing"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750187    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="4abd92cb-973c-4d9d-9a7e-003109eef888" containerName="main"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750195    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f074459-f167-446e-9acf-075cdbfc4ef0" containerName="payload-pre-processing"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750204    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcd47c56-cc43-41ff-951b-82c5768ad799" containerName="payload-pre-processing"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750212    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa825357-78e3-4507-884b-b071ae001d58" containerName="maas-api"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750220    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6e7fd23-0f9d-4ec7-9a57-1326c2a63287" containerName="maas-api"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750227    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad241f3c-90c0-4f0f-b90c-47eec0af0245" containerName="istio-proxy"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750235    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="afdf3167-09b7-498c-8674-31b63af5622c" containerName="maas-api"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750243    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd24563c-4cba-4ad1-9f2b-0f62b199634f" containerName="main"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750250    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0672d75-59ba-490f-bc47-7b2f37ad0e1f" containerName="payload-processing"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750259    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4b81cef-71a5-4458-97c9-f617fc049a8b" containerName="maas-api"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750265    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6ba69b5-5aac-45b5-a324-1557e64456e8" containerName="istio-proxy"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750273    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="feb6a504-7873-440d-a013-36654e702307" containerName="payload-processing"
Nov 03 21:06:10.752641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.750281    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="188223ef-2d2c-4895-9f16-98b1b45aaf6f" containerName="main"
Nov 03 21:06:10.754680 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.754662    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:10.763211 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.763185    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4deb62c4-openshift-default-dockercfg-ft779\""
Nov 03 21:06:10.797481 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.797443    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"]
Nov 03 21:06:10.904205 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.904166    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-data\") pod \"e2e-ait-4deb62c4-openshift-default-794b645577-9plzn\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") " pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:10.904205 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.904207    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2frz\" (UniqueName: \"kubernetes.io/projected/3b3f9906-f91a-49ea-8fd7-64062fde37a8-kube-api-access-r2frz\") pod \"e2e-ait-4deb62c4-openshift-default-794b645577-9plzn\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") " pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:10.904423 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.904231    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-envoy\") pod \"e2e-ait-4deb62c4-openshift-default-794b645577-9plzn\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") " pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:10.904423 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.904288    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-workload-certs\") pod \"e2e-ait-4deb62c4-openshift-default-794b645577-9plzn\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") " pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:10.904497 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.904414    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-podinfo\") pod \"e2e-ait-4deb62c4-openshift-default-794b645577-9plzn\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") " pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:10.904497 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.904462    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-token\") pod \"e2e-ait-4deb62c4-openshift-default-794b645577-9plzn\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") " pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:10.904565 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.904512    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-workload-socket\") pod \"e2e-ait-4deb62c4-openshift-default-794b645577-9plzn\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") " pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:10.904565 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.904543    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-credential-socket\") pod \"e2e-ait-4deb62c4-openshift-default-794b645577-9plzn\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") " pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:10.904665 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:10.904626    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istiod-ca-cert\") pod \"e2e-ait-4deb62c4-openshift-default-794b645577-9plzn\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") " pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:11.006080 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:11.005999    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-podinfo\") pod \"e2e-ait-4deb62c4-openshift-default-794b645577-9plzn\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") " pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:11.006080 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:11.006037    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-token\") pod \"e2e-ait-4deb62c4-openshift-default-794b645577-9plzn\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") " pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:11.006080 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:11.006076    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-workload-socket\") pod \"e2e-ait-4deb62c4-openshift-default-794b645577-9plzn\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") " pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:11.006359 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:11.006103    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-credential-socket\") pod \"e2e-ait-4deb62c4-openshift-default-794b645577-9plzn\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") " pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:11.006359 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:11.006125    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istiod-ca-cert\") pod \"e2e-ait-4deb62c4-openshift-default-794b645577-9plzn\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") " pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:11.006359 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:11.006200    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-data\") pod \"e2e-ait-4deb62c4-openshift-default-794b645577-9plzn\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") " pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:11.006359 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:11.006223    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2frz\" (UniqueName: \"kubernetes.io/projected/3b3f9906-f91a-49ea-8fd7-64062fde37a8-kube-api-access-r2frz\") pod \"e2e-ait-4deb62c4-openshift-default-794b645577-9plzn\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") " pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:11.006359 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:11.006248    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-envoy\") pod \"e2e-ait-4deb62c4-openshift-default-794b645577-9plzn\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") " pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:11.006359 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:11.006269    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-workload-certs\") pod \"e2e-ait-4deb62c4-openshift-default-794b645577-9plzn\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") " pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:11.006738 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:11.006632    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-workload-socket\") pod \"e2e-ait-4deb62c4-openshift-default-794b645577-9plzn\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") " pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:11.006738 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:11.006649    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-credential-socket\") pod \"e2e-ait-4deb62c4-openshift-default-794b645577-9plzn\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") " pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:11.006738 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:11.006663    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-workload-certs\") pod \"e2e-ait-4deb62c4-openshift-default-794b645577-9plzn\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") " pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:11.006913 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:11.006744    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-data\") pod \"e2e-ait-4deb62c4-openshift-default-794b645577-9plzn\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") " pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:11.007256 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:11.007232    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istiod-ca-cert\") pod \"e2e-ait-4deb62c4-openshift-default-794b645577-9plzn\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") " pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:11.008771 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:11.008752    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-podinfo\") pod \"e2e-ait-4deb62c4-openshift-default-794b645577-9plzn\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") " pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:11.008880 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:11.008752    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-envoy\") pod \"e2e-ait-4deb62c4-openshift-default-794b645577-9plzn\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") " pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:11.019804 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:11.019777    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-token\") pod \"e2e-ait-4deb62c4-openshift-default-794b645577-9plzn\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") " pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:11.019991 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:11.019974    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2frz\" (UniqueName: \"kubernetes.io/projected/3b3f9906-f91a-49ea-8fd7-64062fde37a8-kube-api-access-r2frz\") pod \"e2e-ait-4deb62c4-openshift-default-794b645577-9plzn\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") " pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:11.068180 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:11.068144    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:11.349399 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:11.349372    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"]
Nov 03 21:06:11.351337 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:06:11.351312    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b3f9906_f91a_49ea_8fd7_64062fde37a8.slice/crio-b0d379940c5b8cc5b5d99f5588655a757ef65b172d11228aa1d6d6da830b6478 WatchSource:0}: Error finding container b0d379940c5b8cc5b5d99f5588655a757ef65b172d11228aa1d6d6da830b6478: Status 404 returned error can't find the container with id b0d379940c5b8cc5b5d99f5588655a757ef65b172d11228aa1d6d6da830b6478
Nov 03 21:06:11.353476 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:11.353441    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:06:11.353585 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:11.353517    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:06:11.353585 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:11.353561    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:06:12.150413 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:12.150381    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn" event={"ID":"3b3f9906-f91a-49ea-8fd7-64062fde37a8","Type":"ContainerStarted","Data":"376915620e8b5c46c03cb0f6704583cd53ed05cc56009235e48c604cc7c740d3"}
Nov 03 21:06:12.150841 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:12.150437    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn" event={"ID":"3b3f9906-f91a-49ea-8fd7-64062fde37a8","Type":"ContainerStarted","Data":"b0d379940c5b8cc5b5d99f5588655a757ef65b172d11228aa1d6d6da830b6478"}
Nov 03 21:06:12.177354 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:12.177299    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn" podStartSLOduration=2.177285397 podStartE2EDuration="2.177285397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:12.174526059 +0000 UTC m=+3902.307654525" watchObservedRunningTime="2026-11-03 21:06:12.177285397 +0000 UTC m=+3902.310413859"
Nov 03 21:06:13.069250 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:13.069215    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:13.074540 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:13.074519    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:13.154730 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:13.154700    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:13.155526 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:13.155506    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:13.611993 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:13.611966    4421 scope.go:122] "RemoveContainer" containerID="900d2c38acb4a4b584714b8f48c7e4b38bacec834cca965e46de926539afacf1"
Nov 03 21:06:14.734811 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:14.734772    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c"]
Nov 03 21:06:14.735353 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:14.735338    4421 cpu_manager.go:414] "containerMap: removing container" podUID="d366ab06-4688-4c4b-b71b-66bb5d20167d" containerName="istio-proxy"
Nov 03 21:06:14.735401 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:14.735355    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d366ab06-4688-4c4b-b71b-66bb5d20167d" containerName="istio-proxy"
Nov 03 21:06:14.735478 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:14.735467    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="56233474-fdc7-4bcb-aa2a-db87ac074168" containerName="istio-proxy"
Nov 03 21:06:14.735512 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:14.735487    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d366ab06-4688-4c4b-b71b-66bb5d20167d" containerName="istio-proxy"
Nov 03 21:06:14.738895 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:14.738873    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c"
Nov 03 21:06:14.748532 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:14.747209    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4deb62c4-serving-cert\""
Nov 03 21:06:14.749777 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:14.749752    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c"]
Nov 03 21:06:14.844584 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:14.844548    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ffcb\" (UniqueName: \"kubernetes.io/projected/0ee4d6be-c53e-4b29-bf0c-e91c35295544-kube-api-access-7ffcb\") pod \"maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c\" (UID: \"0ee4d6be-c53e-4b29-bf0c-e91c35295544\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c"
Nov 03 21:06:14.844584 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:14.844585    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ee4d6be-c53e-4b29-bf0c-e91c35295544-maas-api-tls\") pod \"maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c\" (UID: \"0ee4d6be-c53e-4b29-bf0c-e91c35295544\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c"
Nov 03 21:06:14.844848 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:14.844757    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ee4d6be-c53e-4b29-bf0c-e91c35295544-metrics-certs\") pod \"maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c\" (UID: \"0ee4d6be-c53e-4b29-bf0c-e91c35295544\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c"
Nov 03 21:06:14.844848 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:14.844821    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ee4d6be-c53e-4b29-bf0c-e91c35295544-openshift-service-ca\") pod \"maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c\" (UID: \"0ee4d6be-c53e-4b29-bf0c-e91c35295544\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c"
Nov 03 21:06:14.938964 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:14.938930    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6"]
Nov 03 21:06:14.942494 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:14.942473    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6"
Nov 03 21:06:14.946392 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:14.946352    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ee4d6be-c53e-4b29-bf0c-e91c35295544-maas-api-tls\") pod \"maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c\" (UID: \"0ee4d6be-c53e-4b29-bf0c-e91c35295544\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c"
Nov 03 21:06:14.946630 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:14.946447    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ee4d6be-c53e-4b29-bf0c-e91c35295544-metrics-certs\") pod \"maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c\" (UID: \"0ee4d6be-c53e-4b29-bf0c-e91c35295544\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c"
Nov 03 21:06:14.946630 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:14.946531    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ee4d6be-c53e-4b29-bf0c-e91c35295544-openshift-service-ca\") pod \"maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c\" (UID: \"0ee4d6be-c53e-4b29-bf0c-e91c35295544\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c"
Nov 03 21:06:14.946630 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:14.946566    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ffcb\" (UniqueName: \"kubernetes.io/projected/0ee4d6be-c53e-4b29-bf0c-e91c35295544-kube-api-access-7ffcb\") pod \"maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c\" (UID: \"0ee4d6be-c53e-4b29-bf0c-e91c35295544\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c"
Nov 03 21:06:14.947985 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:14.947419    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ee4d6be-c53e-4b29-bf0c-e91c35295544-openshift-service-ca\") pod \"maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c\" (UID: \"0ee4d6be-c53e-4b29-bf0c-e91c35295544\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c"
Nov 03 21:06:14.951037 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:14.950314    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ee4d6be-c53e-4b29-bf0c-e91c35295544-metrics-certs\") pod \"maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c\" (UID: \"0ee4d6be-c53e-4b29-bf0c-e91c35295544\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c"
Nov 03 21:06:14.951805 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:14.951780    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ee4d6be-c53e-4b29-bf0c-e91c35295544-maas-api-tls\") pod \"maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c\" (UID: \"0ee4d6be-c53e-4b29-bf0c-e91c35295544\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c"
Nov 03 21:06:14.953180 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:14.953155    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6"]
Nov 03 21:06:14.962301 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:14.962272    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ffcb\" (UniqueName: \"kubernetes.io/projected/0ee4d6be-c53e-4b29-bf0c-e91c35295544-kube-api-access-7ffcb\") pod \"maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c\" (UID: \"0ee4d6be-c53e-4b29-bf0c-e91c35295544\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c"
Nov 03 21:06:14.985848 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:14.985728    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4deb62c4-dockercfg-6qd4t\""
Nov 03 21:06:15.019838 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:15.019806    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4deb62c4\""
Nov 03 21:06:15.047459 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:15.047424    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd64ed2d-29f4-4620-8d22-5a4dcb3ec287-plugins-config-volume\") pod \"payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6\" (UID: \"cd64ed2d-29f4-4620-8d22-5a4dcb3ec287\") " pod="openshift-ingress/payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6"
Nov 03 21:06:15.047651 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:15.047545    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mphp\" (UniqueName: \"kubernetes.io/projected/cd64ed2d-29f4-4620-8d22-5a4dcb3ec287-kube-api-access-8mphp\") pod \"payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6\" (UID: \"cd64ed2d-29f4-4620-8d22-5a4dcb3ec287\") " pod="openshift-ingress/payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6"
Nov 03 21:06:15.057262 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:15.057239    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c"
Nov 03 21:06:15.148421 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:15.148389    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd64ed2d-29f4-4620-8d22-5a4dcb3ec287-plugins-config-volume\") pod \"payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6\" (UID: \"cd64ed2d-29f4-4620-8d22-5a4dcb3ec287\") " pod="openshift-ingress/payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6"
Nov 03 21:06:15.148580 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:15.148473    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mphp\" (UniqueName: \"kubernetes.io/projected/cd64ed2d-29f4-4620-8d22-5a4dcb3ec287-kube-api-access-8mphp\") pod \"payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6\" (UID: \"cd64ed2d-29f4-4620-8d22-5a4dcb3ec287\") " pod="openshift-ingress/payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6"
Nov 03 21:06:15.149256 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:15.149236    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd64ed2d-29f4-4620-8d22-5a4dcb3ec287-plugins-config-volume\") pod \"payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6\" (UID: \"cd64ed2d-29f4-4620-8d22-5a4dcb3ec287\") " pod="openshift-ingress/payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6"
Nov 03 21:06:15.160855 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:15.160829    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mphp\" (UniqueName: \"kubernetes.io/projected/cd64ed2d-29f4-4620-8d22-5a4dcb3ec287-kube-api-access-8mphp\") pod \"payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6\" (UID: \"cd64ed2d-29f4-4620-8d22-5a4dcb3ec287\") " pod="openshift-ingress/payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6"
Nov 03 21:06:15.277977 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:15.277885    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6"
Nov 03 21:06:15.335742 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:15.335244    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c"]
Nov 03 21:06:15.341092 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:06:15.341053    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee4d6be_c53e_4b29_bf0c_e91c35295544.slice/crio-ec89dbcbc2f2a7a9f832592418b5f88329243ff67abc33aa3a54f39e3926acaf WatchSource:0}: Error finding container ec89dbcbc2f2a7a9f832592418b5f88329243ff67abc33aa3a54f39e3926acaf: Status 404 returned error can't find the container with id ec89dbcbc2f2a7a9f832592418b5f88329243ff67abc33aa3a54f39e3926acaf
Nov 03 21:06:15.343756 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:15.342839    4421 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:06:15.560725 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:15.560694    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6"]
Nov 03 21:06:15.563337 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:06:15.563307    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd64ed2d_29f4_4620_8d22_5a4dcb3ec287.slice/crio-e417777e5abe2f572bfb319217ef884f28ebd480cbfdcca5e189e086896248cf WatchSource:0}: Error finding container e417777e5abe2f572bfb319217ef884f28ebd480cbfdcca5e189e086896248cf: Status 404 returned error can't find the container with id e417777e5abe2f572bfb319217ef884f28ebd480cbfdcca5e189e086896248cf
Nov 03 21:06:16.179325 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:16.179226    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6" event={"ID":"cd64ed2d-29f4-4620-8d22-5a4dcb3ec287","Type":"ContainerStarted","Data":"2bc5aa949fcabbb4c28b6b60fe869dff31229c5373f09c1fb2214128797fe09b"}
Nov 03 21:06:16.179810 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:16.179376    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6"
Nov 03 21:06:16.179810 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:16.179402    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6" event={"ID":"cd64ed2d-29f4-4620-8d22-5a4dcb3ec287","Type":"ContainerStarted","Data":"e417777e5abe2f572bfb319217ef884f28ebd480cbfdcca5e189e086896248cf"}
Nov 03 21:06:16.180785 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:16.180764    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c" event={"ID":"0ee4d6be-c53e-4b29-bf0c-e91c35295544","Type":"ContainerStarted","Data":"4ba7774f3729417f8082108976eef0c20654e5b0fcf1c7d7640dd2724aa5441a"}
Nov 03 21:06:16.180859 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:16.180800    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c" event={"ID":"0ee4d6be-c53e-4b29-bf0c-e91c35295544","Type":"ContainerStarted","Data":"ec89dbcbc2f2a7a9f832592418b5f88329243ff67abc33aa3a54f39e3926acaf"}
Nov 03 21:06:16.180859 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:16.180854    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c"
Nov 03 21:06:16.199745 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:16.199688    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6" podStartSLOduration=2.199671692 podStartE2EDuration="2.199671692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:16.197241965 +0000 UTC m=+3906.330370427" watchObservedRunningTime="2026-11-03 21:06:16.199671692 +0000 UTC m=+3906.332800156"
Nov 03 21:06:16.224892 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:16.224846    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c" podStartSLOduration=1.717139602 podStartE2EDuration="2.224832781s" totalImagesPullingTime="507.693179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:16.222361624 +0000 UTC m=+3906.355490081" watchObservedRunningTime="2026-11-03 21:06:16.224832781 +0000 UTC m=+3906.357961242"
Nov 03 21:06:19.628915 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:06:19.628822    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:06:22.190938 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:22.190908    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c"
Nov 03 21:06:27.189319 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:27.189290    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6"
Nov 03 21:06:27.232241 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:27.232206    4421 generic.go:356] "Generic (PLEG): container finished" podID="3b3f9906-f91a-49ea-8fd7-64062fde37a8" containerID="376915620e8b5c46c03cb0f6704583cd53ed05cc56009235e48c604cc7c740d3" exitCode=0
Nov 03 21:06:27.232445 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:27.232245    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn" event={"ID":"3b3f9906-f91a-49ea-8fd7-64062fde37a8","Type":"ContainerDied","Data":"376915620e8b5c46c03cb0f6704583cd53ed05cc56009235e48c604cc7c740d3"}
Nov 03 21:06:27.232726 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:27.232706    4421 scope.go:122] "RemoveContainer" containerID="376915620e8b5c46c03cb0f6704583cd53ed05cc56009235e48c604cc7c740d3"
Nov 03 21:06:27.234101 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:27.234062    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:06:27.234192 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:27.234171    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:06:27.234261 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:27.234240    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:06:28.156744 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:28.156706    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:28.238463 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:28.238427    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn" event={"ID":"3b3f9906-f91a-49ea-8fd7-64062fde37a8","Type":"ContainerStarted","Data":"a875dc4aed3e288f1d41dd655a860d16a9bc437d197a4f6dcad45a541e1023ee"}
Nov 03 21:06:29.068814 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:29.068775    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:29.070156 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:29.070124    4421 patch_prober.go:29] interesting pod/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body=
Nov 03 21:06:29.070311 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:29.070189    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn" podUID="3b3f9906-f91a-49ea-8fd7-64062fde37a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused"
Nov 03 21:06:30.069200 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:30.069146    4421 patch_prober.go:29] interesting pod/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body=
Nov 03 21:06:30.069738 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:30.069239    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn" podUID="3b3f9906-f91a-49ea-8fd7-64062fde37a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused"
Nov 03 21:06:31.068861 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:31.068821    4421 patch_prober.go:29] interesting pod/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body=
Nov 03 21:06:31.069087 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:31.068888    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn" podUID="3b3f9906-f91a-49ea-8fd7-64062fde37a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused"
Nov 03 21:06:32.069079 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:32.069032    4421 patch_prober.go:29] interesting pod/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body=
Nov 03 21:06:32.069644 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:32.069094    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn" podUID="3b3f9906-f91a-49ea-8fd7-64062fde37a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused"
Nov 03 21:06:33.069272 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:33.069231    4421 patch_prober.go:29] interesting pod/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body=
Nov 03 21:06:33.069738 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:33.069293    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn" podUID="3b3f9906-f91a-49ea-8fd7-64062fde37a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused"
Nov 03 21:06:34.073183 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:34.073146    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:34.073648 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:34.073625    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:34.074336 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:34.074316    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:39.729961 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:39.729923    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6"]
Nov 03 21:06:39.904322 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:39.904282    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c"]
Nov 03 21:06:39.904554 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:39.904521    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c" podUID="0ee4d6be-c53e-4b29-bf0c-e91c35295544" containerName="maas-api" containerID="cri-o://4ba7774f3729417f8082108976eef0c20654e5b0fcf1c7d7640dd2724aa5441a" gracePeriod=30
Nov 03 21:06:40.168493 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:40.168461    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c"
Nov 03 21:06:40.311343 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:40.311259    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ee4d6be-c53e-4b29-bf0c-e91c35295544-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ee4d6be-c53e-4b29-bf0c-e91c35295544-openshift-service-ca\") pod \"0ee4d6be-c53e-4b29-bf0c-e91c35295544\" (UID: \"0ee4d6be-c53e-4b29-bf0c-e91c35295544\") "
Nov 03 21:06:40.311343 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:40.311313    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ee4d6be-c53e-4b29-bf0c-e91c35295544-kube-api-access-7ffcb\" (UniqueName: \"kubernetes.io/projected/0ee4d6be-c53e-4b29-bf0c-e91c35295544-kube-api-access-7ffcb\") pod \"0ee4d6be-c53e-4b29-bf0c-e91c35295544\" (UID: \"0ee4d6be-c53e-4b29-bf0c-e91c35295544\") "
Nov 03 21:06:40.311586 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:40.311475    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ee4d6be-c53e-4b29-bf0c-e91c35295544-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ee4d6be-c53e-4b29-bf0c-e91c35295544-metrics-certs\") pod \"0ee4d6be-c53e-4b29-bf0c-e91c35295544\" (UID: \"0ee4d6be-c53e-4b29-bf0c-e91c35295544\") "
Nov 03 21:06:40.311677 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:40.311619    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ee4d6be-c53e-4b29-bf0c-e91c35295544-openshift-service-ca" pod "0ee4d6be-c53e-4b29-bf0c-e91c35295544" (UID: "0ee4d6be-c53e-4b29-bf0c-e91c35295544"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:06:40.311783 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:40.311756    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ee4d6be-c53e-4b29-bf0c-e91c35295544-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ee4d6be-c53e-4b29-bf0c-e91c35295544-maas-api-tls\") pod \"0ee4d6be-c53e-4b29-bf0c-e91c35295544\" (UID: \"0ee4d6be-c53e-4b29-bf0c-e91c35295544\") "
Nov 03 21:06:40.312436 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:40.312413    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ee4d6be-c53e-4b29-bf0c-e91c35295544-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:40.314592 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:40.314552    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ee4d6be-c53e-4b29-bf0c-e91c35295544-metrics-certs" pod "0ee4d6be-c53e-4b29-bf0c-e91c35295544" (UID: "0ee4d6be-c53e-4b29-bf0c-e91c35295544"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:06:40.314592 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:40.314582    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee4d6be-c53e-4b29-bf0c-e91c35295544-kube-api-access-7ffcb" pod "0ee4d6be-c53e-4b29-bf0c-e91c35295544" (UID: "0ee4d6be-c53e-4b29-bf0c-e91c35295544"). InnerVolumeSpecName "kube-api-access-7ffcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:06:40.316940 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:40.316900    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ee4d6be-c53e-4b29-bf0c-e91c35295544-maas-api-tls" pod "0ee4d6be-c53e-4b29-bf0c-e91c35295544" (UID: "0ee4d6be-c53e-4b29-bf0c-e91c35295544"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:06:40.319125 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:40.319102    4421 generic.go:356] "Generic (PLEG): container finished" podID="0ee4d6be-c53e-4b29-bf0c-e91c35295544" containerID="4ba7774f3729417f8082108976eef0c20654e5b0fcf1c7d7640dd2724aa5441a" exitCode=0
Nov 03 21:06:40.319235 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:40.319158    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c"
Nov 03 21:06:40.319235 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:40.319195    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c" event={"ID":"0ee4d6be-c53e-4b29-bf0c-e91c35295544","Type":"ContainerDied","Data":"4ba7774f3729417f8082108976eef0c20654e5b0fcf1c7d7640dd2724aa5441a"}
Nov 03 21:06:40.319329 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:40.319271    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c" event={"ID":"0ee4d6be-c53e-4b29-bf0c-e91c35295544","Type":"ContainerDied","Data":"ec89dbcbc2f2a7a9f832592418b5f88329243ff67abc33aa3a54f39e3926acaf"}
Nov 03 21:06:40.319329 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:40.319305    4421 scope.go:122] "RemoveContainer" containerID="4ba7774f3729417f8082108976eef0c20654e5b0fcf1c7d7640dd2724aa5441a"
Nov 03 21:06:40.335632 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:40.335591    4421 scope.go:122] "RemoveContainer" containerID="4ba7774f3729417f8082108976eef0c20654e5b0fcf1c7d7640dd2724aa5441a"
Nov 03 21:06:40.336086 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:06:40.335987    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ba7774f3729417f8082108976eef0c20654e5b0fcf1c7d7640dd2724aa5441a\": container with ID starting with 4ba7774f3729417f8082108976eef0c20654e5b0fcf1c7d7640dd2724aa5441a not found: ID does not exist" containerID="4ba7774f3729417f8082108976eef0c20654e5b0fcf1c7d7640dd2724aa5441a"
Nov 03 21:06:40.336086 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:40.336026    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba7774f3729417f8082108976eef0c20654e5b0fcf1c7d7640dd2724aa5441a"} err="failed to get container status \"4ba7774f3729417f8082108976eef0c20654e5b0fcf1c7d7640dd2724aa5441a\": rpc error: code = NotFound desc = could not find container \"4ba7774f3729417f8082108976eef0c20654e5b0fcf1c7d7640dd2724aa5441a\": container with ID starting with 4ba7774f3729417f8082108976eef0c20654e5b0fcf1c7d7640dd2724aa5441a not found: ID does not exist"
Nov 03 21:06:40.345924 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:40.345876    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c"]
Nov 03 21:06:40.351940 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:40.351918    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4deb62c4-7444bd55bf-mbh2c"]
Nov 03 21:06:40.413675 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:40.413640    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ffcb\" (UniqueName: \"kubernetes.io/projected/0ee4d6be-c53e-4b29-bf0c-e91c35295544-kube-api-access-7ffcb\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:40.413675 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:40.413669    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ee4d6be-c53e-4b29-bf0c-e91c35295544-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:40.413675 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:40.413680    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ee4d6be-c53e-4b29-bf0c-e91c35295544-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:42.014387 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:42.014352    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ee4d6be-c53e-4b29-bf0c-e91c35295544" path="/var/lib/kubelet/pods/0ee4d6be-c53e-4b29-bf0c-e91c35295544/volumes"
Nov 03 21:06:44.730260 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:44.730221    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6" podUID="cd64ed2d-29f4-4620-8d22-5a4dcb3ec287" containerName="payload-processing" containerID="cri-o://2bc5aa949fcabbb4c28b6b60fe869dff31229c5373f09c1fb2214128797fe09b" gracePeriod=25
Nov 03 21:06:45.006724 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:45.006697    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6"
Nov 03 21:06:45.182439 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:45.182410    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd64ed2d-29f4-4620-8d22-5a4dcb3ec287-kube-api-access-8mphp\" (UniqueName: \"kubernetes.io/projected/cd64ed2d-29f4-4620-8d22-5a4dcb3ec287-kube-api-access-8mphp\") pod \"cd64ed2d-29f4-4620-8d22-5a4dcb3ec287\" (UID: \"cd64ed2d-29f4-4620-8d22-5a4dcb3ec287\") "
Nov 03 21:06:45.182668 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:45.182496    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd64ed2d-29f4-4620-8d22-5a4dcb3ec287-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd64ed2d-29f4-4620-8d22-5a4dcb3ec287-plugins-config-volume\") pod \"cd64ed2d-29f4-4620-8d22-5a4dcb3ec287\" (UID: \"cd64ed2d-29f4-4620-8d22-5a4dcb3ec287\") "
Nov 03 21:06:45.182952 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:45.182924    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd64ed2d-29f4-4620-8d22-5a4dcb3ec287-plugins-config-volume" pod "cd64ed2d-29f4-4620-8d22-5a4dcb3ec287" (UID: "cd64ed2d-29f4-4620-8d22-5a4dcb3ec287"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:06:45.184676 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:45.184650    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd64ed2d-29f4-4620-8d22-5a4dcb3ec287-kube-api-access-8mphp" pod "cd64ed2d-29f4-4620-8d22-5a4dcb3ec287" (UID: "cd64ed2d-29f4-4620-8d22-5a4dcb3ec287"). InnerVolumeSpecName "kube-api-access-8mphp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:06:45.283652 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:45.283561    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mphp\" (UniqueName: \"kubernetes.io/projected/cd64ed2d-29f4-4620-8d22-5a4dcb3ec287-kube-api-access-8mphp\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:45.283652 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:45.283592    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd64ed2d-29f4-4620-8d22-5a4dcb3ec287-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:45.347532 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:45.347496    4421 generic.go:356] "Generic (PLEG): container finished" podID="cd64ed2d-29f4-4620-8d22-5a4dcb3ec287" containerID="2bc5aa949fcabbb4c28b6b60fe869dff31229c5373f09c1fb2214128797fe09b" exitCode=0
Nov 03 21:06:45.347710 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:45.347541    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6" event={"ID":"cd64ed2d-29f4-4620-8d22-5a4dcb3ec287","Type":"ContainerDied","Data":"2bc5aa949fcabbb4c28b6b60fe869dff31229c5373f09c1fb2214128797fe09b"}
Nov 03 21:06:45.347710 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:45.347549    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6"
Nov 03 21:06:45.347710 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:45.347611    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6" event={"ID":"cd64ed2d-29f4-4620-8d22-5a4dcb3ec287","Type":"ContainerDied","Data":"e417777e5abe2f572bfb319217ef884f28ebd480cbfdcca5e189e086896248cf"}
Nov 03 21:06:45.347710 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:45.347637    4421 scope.go:122] "RemoveContainer" containerID="2bc5aa949fcabbb4c28b6b60fe869dff31229c5373f09c1fb2214128797fe09b"
Nov 03 21:06:45.356433 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:45.356414    4421 scope.go:122] "RemoveContainer" containerID="2bc5aa949fcabbb4c28b6b60fe869dff31229c5373f09c1fb2214128797fe09b"
Nov 03 21:06:45.356806 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:06:45.356786    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bc5aa949fcabbb4c28b6b60fe869dff31229c5373f09c1fb2214128797fe09b\": container with ID starting with 2bc5aa949fcabbb4c28b6b60fe869dff31229c5373f09c1fb2214128797fe09b not found: ID does not exist" containerID="2bc5aa949fcabbb4c28b6b60fe869dff31229c5373f09c1fb2214128797fe09b"
Nov 03 21:06:45.356890 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:45.356815    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bc5aa949fcabbb4c28b6b60fe869dff31229c5373f09c1fb2214128797fe09b"} err="failed to get container status \"2bc5aa949fcabbb4c28b6b60fe869dff31229c5373f09c1fb2214128797fe09b\": rpc error: code = NotFound desc = could not find container \"2bc5aa949fcabbb4c28b6b60fe869dff31229c5373f09c1fb2214128797fe09b\": container with ID starting with 2bc5aa949fcabbb4c28b6b60fe869dff31229c5373f09c1fb2214128797fe09b not found: ID does not exist"
Nov 03 21:06:45.369731 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:45.369707    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6"]
Nov 03 21:06:45.374854 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:45.374831    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4deb62c4-57bbb6f6db-jxhx6"]
Nov 03 21:06:46.014788 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:46.014753    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd64ed2d-29f4-4620-8d22-5a4dcb3ec287" path="/var/lib/kubelet/pods/cd64ed2d-29f4-4620-8d22-5a4dcb3ec287/volumes"
Nov 03 21:06:46.028719 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:46.028687    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"]
Nov 03 21:06:46.029005 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:46.028977    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn" podUID="3b3f9906-f91a-49ea-8fd7-64062fde37a8" containerName="istio-proxy" containerID="cri-o://a875dc4aed3e288f1d41dd655a860d16a9bc437d197a4f6dcad45a541e1023ee" gracePeriod=30
Nov 03 21:06:47.678483 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:47.678447    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-687b69bd9-6n6dr"]
Nov 03 21:06:47.678994 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:47.678655    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-687b69bd9-6n6dr" podUID="2bd8fabe-6f14-458d-ae20-bc4b71333adf" containerName="manager" containerID="cri-o://320f29b71fa69d9f2d721bcc3281c3f9c573e68e059a49d5f5cb9f49bca74437" gracePeriod=30
Nov 03 21:06:47.925574 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:47.925546    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-687b69bd9-6n6dr"
Nov 03 21:06:48.111753 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:48.111723    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd8fabe-6f14-458d-ae20-bc4b71333adf-kube-api-access-v9pt2\" (UniqueName: \"kubernetes.io/projected/2bd8fabe-6f14-458d-ae20-bc4b71333adf-kube-api-access-v9pt2\") pod \"2bd8fabe-6f14-458d-ae20-bc4b71333adf\" (UID: \"2bd8fabe-6f14-458d-ae20-bc4b71333adf\") "
Nov 03 21:06:48.111965 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:48.111819    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bd8fabe-6f14-458d-ae20-bc4b71333adf-webhook-cert\" (UniqueName: \"kubernetes.io/secret/2bd8fabe-6f14-458d-ae20-bc4b71333adf-webhook-cert\") pod \"2bd8fabe-6f14-458d-ae20-bc4b71333adf\" (UID: \"2bd8fabe-6f14-458d-ae20-bc4b71333adf\") "
Nov 03 21:06:48.111965 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:48.111883    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bd8fabe-6f14-458d-ae20-bc4b71333adf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd8fabe-6f14-458d-ae20-bc4b71333adf-metrics-certs\") pod \"2bd8fabe-6f14-458d-ae20-bc4b71333adf\" (UID: \"2bd8fabe-6f14-458d-ae20-bc4b71333adf\") "
Nov 03 21:06:48.114032 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:48.113999    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd8fabe-6f14-458d-ae20-bc4b71333adf-kube-api-access-v9pt2" pod "2bd8fabe-6f14-458d-ae20-bc4b71333adf" (UID: "2bd8fabe-6f14-458d-ae20-bc4b71333adf"). InnerVolumeSpecName "kube-api-access-v9pt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:06:48.114171 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:48.114050    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd8fabe-6f14-458d-ae20-bc4b71333adf-metrics-certs" pod "2bd8fabe-6f14-458d-ae20-bc4b71333adf" (UID: "2bd8fabe-6f14-458d-ae20-bc4b71333adf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:06:48.114171 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:48.114065    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd8fabe-6f14-458d-ae20-bc4b71333adf-webhook-cert" pod "2bd8fabe-6f14-458d-ae20-bc4b71333adf" (UID: "2bd8fabe-6f14-458d-ae20-bc4b71333adf"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:06:48.213415 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:48.213381    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9pt2\" (UniqueName: \"kubernetes.io/projected/2bd8fabe-6f14-458d-ae20-bc4b71333adf-kube-api-access-v9pt2\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:48.213415 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:48.213411    4421 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2bd8fabe-6f14-458d-ae20-bc4b71333adf-webhook-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:48.213415 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:48.213421    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bd8fabe-6f14-458d-ae20-bc4b71333adf-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:48.362632 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:48.362525    4421 generic.go:356] "Generic (PLEG): container finished" podID="2bd8fabe-6f14-458d-ae20-bc4b71333adf" containerID="320f29b71fa69d9f2d721bcc3281c3f9c573e68e059a49d5f5cb9f49bca74437" exitCode=0
Nov 03 21:06:48.362632 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:48.362574    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-687b69bd9-6n6dr"
Nov 03 21:06:48.362840 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:48.362632    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-687b69bd9-6n6dr" event={"ID":"2bd8fabe-6f14-458d-ae20-bc4b71333adf","Type":"ContainerDied","Data":"320f29b71fa69d9f2d721bcc3281c3f9c573e68e059a49d5f5cb9f49bca74437"}
Nov 03 21:06:48.362840 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:48.362687    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-687b69bd9-6n6dr" event={"ID":"2bd8fabe-6f14-458d-ae20-bc4b71333adf","Type":"ContainerDied","Data":"ad5c891e7279b2c3cdab83108c40e12f1fd3a1cc68203c67452325d41852f7c9"}
Nov 03 21:06:48.362840 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:48.362710    4421 scope.go:122] "RemoveContainer" containerID="320f29b71fa69d9f2d721bcc3281c3f9c573e68e059a49d5f5cb9f49bca74437"
Nov 03 21:06:48.374878 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:48.374860    4421 scope.go:122] "RemoveContainer" containerID="320f29b71fa69d9f2d721bcc3281c3f9c573e68e059a49d5f5cb9f49bca74437"
Nov 03 21:06:48.375161 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:06:48.375140    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"320f29b71fa69d9f2d721bcc3281c3f9c573e68e059a49d5f5cb9f49bca74437\": container with ID starting with 320f29b71fa69d9f2d721bcc3281c3f9c573e68e059a49d5f5cb9f49bca74437 not found: ID does not exist" containerID="320f29b71fa69d9f2d721bcc3281c3f9c573e68e059a49d5f5cb9f49bca74437"
Nov 03 21:06:48.375229 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:48.375174    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"320f29b71fa69d9f2d721bcc3281c3f9c573e68e059a49d5f5cb9f49bca74437"} err="failed to get container status \"320f29b71fa69d9f2d721bcc3281c3f9c573e68e059a49d5f5cb9f49bca74437\": rpc error: code = NotFound desc = could not find container \"320f29b71fa69d9f2d721bcc3281c3f9c573e68e059a49d5f5cb9f49bca74437\": container with ID starting with 320f29b71fa69d9f2d721bcc3281c3f9c573e68e059a49d5f5cb9f49bca74437 not found: ID does not exist"
Nov 03 21:06:48.400892 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:48.400862    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-687b69bd9-6n6dr"]
Nov 03 21:06:48.411204 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:48.411177    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-687b69bd9-6n6dr"]
Nov 03 21:06:49.074132 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.074086    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-687b69bd9-q5zvz"]
Nov 03 21:06:49.074686 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.074667    4421 cpu_manager.go:414] "containerMap: removing container" podUID="cd64ed2d-29f4-4620-8d22-5a4dcb3ec287" containerName="payload-processing"
Nov 03 21:06:49.074767 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.074688    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd64ed2d-29f4-4620-8d22-5a4dcb3ec287" containerName="payload-processing"
Nov 03 21:06:49.074767 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.074731    4421 cpu_manager.go:414] "containerMap: removing container" podUID="2bd8fabe-6f14-458d-ae20-bc4b71333adf" containerName="manager"
Nov 03 21:06:49.074767 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.074741    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd8fabe-6f14-458d-ae20-bc4b71333adf" containerName="manager"
Nov 03 21:06:49.074767 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.074752    4421 cpu_manager.go:414] "containerMap: removing container" podUID="0ee4d6be-c53e-4b29-bf0c-e91c35295544" containerName="maas-api"
Nov 03 21:06:49.074767 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.074761    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ee4d6be-c53e-4b29-bf0c-e91c35295544" containerName="maas-api"
Nov 03 21:06:49.075049 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.074863    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd8fabe-6f14-458d-ae20-bc4b71333adf" containerName="manager"
Nov 03 21:06:49.075049 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.074875    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ee4d6be-c53e-4b29-bf0c-e91c35295544" containerName="maas-api"
Nov 03 21:06:49.075049 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.074889    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd64ed2d-29f4-4620-8d22-5a4dcb3ec287" containerName="payload-processing"
Nov 03 21:06:49.076810 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.076774    4421 patch_prober.go:29] interesting pod/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 21:06:49.076996 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.076830    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn" podUID="3b3f9906-f91a-49ea-8fd7-64062fde37a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 21:06:49.079612 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.079581    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-687b69bd9-q5zvz"
Nov 03 21:06:49.088805 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.087859    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 21:06:49.095632 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.091433    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 21:06:49.095632 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.091584    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 21:06:49.095632 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.094722    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-687b69bd9-q5zvz"]
Nov 03 21:06:49.095632 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.094858    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7tsr7\""
Nov 03 21:06:49.222966 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.222924    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggb4b\" (UniqueName: \"kubernetes.io/projected/c66f5c12-d8d2-491d-a836-fc8776ea6261-kube-api-access-ggb4b\") pod \"maas-controller-687b69bd9-q5zvz\" (UID: \"c66f5c12-d8d2-491d-a836-fc8776ea6261\") " pod="opendatahub/maas-controller-687b69bd9-q5zvz"
Nov 03 21:06:49.222966 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.222962    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c66f5c12-d8d2-491d-a836-fc8776ea6261-metrics-certs\") pod \"maas-controller-687b69bd9-q5zvz\" (UID: \"c66f5c12-d8d2-491d-a836-fc8776ea6261\") " pod="opendatahub/maas-controller-687b69bd9-q5zvz"
Nov 03 21:06:49.223212 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.223177    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c66f5c12-d8d2-491d-a836-fc8776ea6261-webhook-cert\") pod \"maas-controller-687b69bd9-q5zvz\" (UID: \"c66f5c12-d8d2-491d-a836-fc8776ea6261\") " pod="opendatahub/maas-controller-687b69bd9-q5zvz"
Nov 03 21:06:49.323737 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.323696    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggb4b\" (UniqueName: \"kubernetes.io/projected/c66f5c12-d8d2-491d-a836-fc8776ea6261-kube-api-access-ggb4b\") pod \"maas-controller-687b69bd9-q5zvz\" (UID: \"c66f5c12-d8d2-491d-a836-fc8776ea6261\") " pod="opendatahub/maas-controller-687b69bd9-q5zvz"
Nov 03 21:06:49.323960 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.323744    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c66f5c12-d8d2-491d-a836-fc8776ea6261-metrics-certs\") pod \"maas-controller-687b69bd9-q5zvz\" (UID: \"c66f5c12-d8d2-491d-a836-fc8776ea6261\") " pod="opendatahub/maas-controller-687b69bd9-q5zvz"
Nov 03 21:06:49.323960 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.323886    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c66f5c12-d8d2-491d-a836-fc8776ea6261-webhook-cert\") pod \"maas-controller-687b69bd9-q5zvz\" (UID: \"c66f5c12-d8d2-491d-a836-fc8776ea6261\") " pod="opendatahub/maas-controller-687b69bd9-q5zvz"
Nov 03 21:06:49.327191 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.327136    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c66f5c12-d8d2-491d-a836-fc8776ea6261-metrics-certs\") pod \"maas-controller-687b69bd9-q5zvz\" (UID: \"c66f5c12-d8d2-491d-a836-fc8776ea6261\") " pod="opendatahub/maas-controller-687b69bd9-q5zvz"
Nov 03 21:06:49.327379 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.327356    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c66f5c12-d8d2-491d-a836-fc8776ea6261-webhook-cert\") pod \"maas-controller-687b69bd9-q5zvz\" (UID: \"c66f5c12-d8d2-491d-a836-fc8776ea6261\") " pod="opendatahub/maas-controller-687b69bd9-q5zvz"
Nov 03 21:06:49.337806 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.337757    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggb4b\" (UniqueName: \"kubernetes.io/projected/c66f5c12-d8d2-491d-a836-fc8776ea6261-kube-api-access-ggb4b\") pod \"maas-controller-687b69bd9-q5zvz\" (UID: \"c66f5c12-d8d2-491d-a836-fc8776ea6261\") " pod="opendatahub/maas-controller-687b69bd9-q5zvz"
Nov 03 21:06:49.399321 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.399285    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-687b69bd9-q5zvz"
Nov 03 21:06:49.628260 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:06:49.628174    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:06:49.681986 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:49.681958    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-687b69bd9-q5zvz"]
Nov 03 21:06:49.683969 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:06:49.683945    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc66f5c12_d8d2_491d_a836_fc8776ea6261.slice/crio-6119e413f344694e0ff43027a6529f78e96cb830b4433a5e3b52770cbf1fb422 WatchSource:0}: Error finding container 6119e413f344694e0ff43027a6529f78e96cb830b4433a5e3b52770cbf1fb422: Status 404 returned error can't find the container with id 6119e413f344694e0ff43027a6529f78e96cb830b4433a5e3b52770cbf1fb422
Nov 03 21:06:50.015342 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:50.015308    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bd8fabe-6f14-458d-ae20-bc4b71333adf" path="/var/lib/kubelet/pods/2bd8fabe-6f14-458d-ae20-bc4b71333adf/volumes"
Nov 03 21:06:50.379937 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:50.379839    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-687b69bd9-q5zvz" event={"ID":"c66f5c12-d8d2-491d-a836-fc8776ea6261","Type":"ContainerStarted","Data":"daa2a342507e607abef7f135face8ff1206b71d56ff7c86ea393efe377823207"}
Nov 03 21:06:50.379937 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:50.379913    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-687b69bd9-q5zvz" event={"ID":"c66f5c12-d8d2-491d-a836-fc8776ea6261","Type":"ContainerStarted","Data":"6119e413f344694e0ff43027a6529f78e96cb830b4433a5e3b52770cbf1fb422"}
Nov 03 21:06:50.380797 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:50.380773    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-687b69bd9-q5zvz"
Nov 03 21:06:50.403213 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:50.403155    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-687b69bd9-q5zvz" podStartSLOduration=0.906111488 podStartE2EDuration="1.403136462s" totalImagesPullingTime="497.024974ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:50.400994196 +0000 UTC m=+3940.534122659" watchObservedRunningTime="2026-11-03 21:06:50.403136462 +0000 UTC m=+3940.536264925"
Nov 03 21:06:51.328161 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.328134    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:51.407823 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.407787    4421 generic.go:356] "Generic (PLEG): container finished" podID="3b3f9906-f91a-49ea-8fd7-64062fde37a8" containerID="a875dc4aed3e288f1d41dd655a860d16a9bc437d197a4f6dcad45a541e1023ee" exitCode=0
Nov 03 21:06:51.408359 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.407846    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"
Nov 03 21:06:51.408359 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.407929    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn" event={"ID":"3b3f9906-f91a-49ea-8fd7-64062fde37a8","Type":"ContainerDied","Data":"a875dc4aed3e288f1d41dd655a860d16a9bc437d197a4f6dcad45a541e1023ee"}
Nov 03 21:06:51.408359 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.408013    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn" event={"ID":"3b3f9906-f91a-49ea-8fd7-64062fde37a8","Type":"ContainerDied","Data":"b0d379940c5b8cc5b5d99f5588655a757ef65b172d11228aa1d6d6da830b6478"}
Nov 03 21:06:51.408359 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.408047    4421 scope.go:122] "RemoveContainer" containerID="a875dc4aed3e288f1d41dd655a860d16a9bc437d197a4f6dcad45a541e1023ee"
Nov 03 21:06:51.416691 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.416674    4421 scope.go:122] "RemoveContainer" containerID="376915620e8b5c46c03cb0f6704583cd53ed05cc56009235e48c604cc7c740d3"
Nov 03 21:06:51.424894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.424863    4421 scope.go:122] "RemoveContainer" containerID="a875dc4aed3e288f1d41dd655a860d16a9bc437d197a4f6dcad45a541e1023ee"
Nov 03 21:06:51.425143 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:06:51.425124    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a875dc4aed3e288f1d41dd655a860d16a9bc437d197a4f6dcad45a541e1023ee\": container with ID starting with a875dc4aed3e288f1d41dd655a860d16a9bc437d197a4f6dcad45a541e1023ee not found: ID does not exist" containerID="a875dc4aed3e288f1d41dd655a860d16a9bc437d197a4f6dcad45a541e1023ee"
Nov 03 21:06:51.425206 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.425154    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a875dc4aed3e288f1d41dd655a860d16a9bc437d197a4f6dcad45a541e1023ee"} err="failed to get container status \"a875dc4aed3e288f1d41dd655a860d16a9bc437d197a4f6dcad45a541e1023ee\": rpc error: code = NotFound desc = could not find container \"a875dc4aed3e288f1d41dd655a860d16a9bc437d197a4f6dcad45a541e1023ee\": container with ID starting with a875dc4aed3e288f1d41dd655a860d16a9bc437d197a4f6dcad45a541e1023ee not found: ID does not exist"
Nov 03 21:06:51.425206 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.425172    4421 scope.go:122] "RemoveContainer" containerID="376915620e8b5c46c03cb0f6704583cd53ed05cc56009235e48c604cc7c740d3"
Nov 03 21:06:51.425443 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:06:51.425419    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"376915620e8b5c46c03cb0f6704583cd53ed05cc56009235e48c604cc7c740d3\": container with ID starting with 376915620e8b5c46c03cb0f6704583cd53ed05cc56009235e48c604cc7c740d3 not found: ID does not exist" containerID="376915620e8b5c46c03cb0f6704583cd53ed05cc56009235e48c604cc7c740d3"
Nov 03 21:06:51.425496 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.425447    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"376915620e8b5c46c03cb0f6704583cd53ed05cc56009235e48c604cc7c740d3"} err="failed to get container status \"376915620e8b5c46c03cb0f6704583cd53ed05cc56009235e48c604cc7c740d3\": rpc error: code = NotFound desc = could not find container \"376915620e8b5c46c03cb0f6704583cd53ed05cc56009235e48c604cc7c740d3\": container with ID starting with 376915620e8b5c46c03cb0f6704583cd53ed05cc56009235e48c604cc7c740d3 not found: ID does not exist"
Nov 03 21:06:51.445653 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.445589    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-workload-socket\") pod \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") "
Nov 03 21:06:51.445804 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.445728    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-envoy\") pod \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") "
Nov 03 21:06:51.445804 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.445767    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-data\") pod \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") "
Nov 03 21:06:51.445804 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.445798    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-credential-socket\") pod \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") "
Nov 03 21:06:51.446015 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.445838    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-podinfo\") pod \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") "
Nov 03 21:06:51.446015 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.445887    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-token\" (UniqueName: \"kubernetes.io/projected/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-token\") pod \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") "
Nov 03 21:06:51.446015 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.445939    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b3f9906-f91a-49ea-8fd7-64062fde37a8-kube-api-access-r2frz\" (UniqueName: \"kubernetes.io/projected/3b3f9906-f91a-49ea-8fd7-64062fde37a8-kube-api-access-r2frz\") pod \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") "
Nov 03 21:06:51.446015 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.446013    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istiod-ca-cert\") pod \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") "
Nov 03 21:06:51.446240 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.446061    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-workload-certs\") pod \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\" (UID: \"3b3f9906-f91a-49ea-8fd7-64062fde37a8\") "
Nov 03 21:06:51.446240 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.445882    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-workload-socket" pod "3b3f9906-f91a-49ea-8fd7-64062fde37a8" (UID: "3b3f9906-f91a-49ea-8fd7-64062fde37a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:06:51.446240 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.446127    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-credential-socket" pod "3b3f9906-f91a-49ea-8fd7-64062fde37a8" (UID: "3b3f9906-f91a-49ea-8fd7-64062fde37a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:06:51.446415 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.446300    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-data" pod "3b3f9906-f91a-49ea-8fd7-64062fde37a8" (UID: "3b3f9906-f91a-49ea-8fd7-64062fde37a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:06:51.446482 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.446428    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:51.446482 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.446450    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:51.446482 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.446465    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:51.447065 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.447016    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-workload-certs" pod "3b3f9906-f91a-49ea-8fd7-64062fde37a8" (UID: "3b3f9906-f91a-49ea-8fd7-64062fde37a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:06:51.447186 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.447157    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istiod-ca-cert" pod "3b3f9906-f91a-49ea-8fd7-64062fde37a8" (UID: "3b3f9906-f91a-49ea-8fd7-64062fde37a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:06:51.449536 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.449500    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-podinfo" pod "3b3f9906-f91a-49ea-8fd7-64062fde37a8" (UID: "3b3f9906-f91a-49ea-8fd7-64062fde37a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:06:51.450120 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.450083    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b3f9906-f91a-49ea-8fd7-64062fde37a8-kube-api-access-r2frz" pod "3b3f9906-f91a-49ea-8fd7-64062fde37a8" (UID: "3b3f9906-f91a-49ea-8fd7-64062fde37a8"). InnerVolumeSpecName "kube-api-access-r2frz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:06:51.450411 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.450372    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-token" pod "3b3f9906-f91a-49ea-8fd7-64062fde37a8" (UID: "3b3f9906-f91a-49ea-8fd7-64062fde37a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:06:51.450411 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.450371    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-envoy" pod "3b3f9906-f91a-49ea-8fd7-64062fde37a8" (UID: "3b3f9906-f91a-49ea-8fd7-64062fde37a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:06:51.547536 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.547386    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:51.547536 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.547423    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:51.547536 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.547438    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:51.547536 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.547454    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2frz\" (UniqueName: \"kubernetes.io/projected/3b3f9906-f91a-49ea-8fd7-64062fde37a8-kube-api-access-r2frz\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:51.547536 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.547469    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b3f9906-f91a-49ea-8fd7-64062fde37a8-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:51.547536 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.547484    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b3f9906-f91a-49ea-8fd7-64062fde37a8-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:06:51.730635 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.730588    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"]
Nov 03 21:06:51.736809 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:51.736785    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4deb62c4-openshift-default-794b645577-9plzn"]
Nov 03 21:06:52.014226 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:06:52.014152    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b3f9906-f91a-49ea-8fd7-64062fde37a8" path="/var/lib/kubelet/pods/3b3f9906-f91a-49ea-8fd7-64062fde37a8/volumes"
Nov 03 21:07:02.415291 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:02.415262    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-687b69bd9-q5zvz"
Nov 03 21:07:03.353724 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.353696    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"]
Nov 03 21:07:03.354176 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.354159    4421 cpu_manager.go:414] "containerMap: removing container" podUID="3b3f9906-f91a-49ea-8fd7-64062fde37a8" containerName="istio-proxy"
Nov 03 21:07:03.354227 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.354179    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b3f9906-f91a-49ea-8fd7-64062fde37a8" containerName="istio-proxy"
Nov 03 21:07:03.354227 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.354189    4421 cpu_manager.go:414] "containerMap: removing container" podUID="3b3f9906-f91a-49ea-8fd7-64062fde37a8" containerName="istio-proxy"
Nov 03 21:07:03.354227 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.354194    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b3f9906-f91a-49ea-8fd7-64062fde37a8" containerName="istio-proxy"
Nov 03 21:07:03.354360 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.354274    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b3f9906-f91a-49ea-8fd7-64062fde37a8" containerName="istio-proxy"
Nov 03 21:07:03.354360 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.354284    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b3f9906-f91a-49ea-8fd7-64062fde37a8" containerName="istio-proxy"
Nov 03 21:07:03.359070 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.359046    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.373176 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.373154    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"]
Nov 03 21:07:03.445921 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.445881    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqkdk\" (UniqueName: \"kubernetes.io/projected/bc25e3fb-91ba-4d75-92d3-d1aab2feb033-kube-api-access-vqkdk\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xfwgr\" (UID: \"bc25e3fb-91ba-4d75-92d3-d1aab2feb033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.446403 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.445936    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc25e3fb-91ba-4d75-92d3-d1aab2feb033-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xfwgr\" (UID: \"bc25e3fb-91ba-4d75-92d3-d1aab2feb033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.446403 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.446042    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc25e3fb-91ba-4d75-92d3-d1aab2feb033-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xfwgr\" (UID: \"bc25e3fb-91ba-4d75-92d3-d1aab2feb033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.446403 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.446069    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc25e3fb-91ba-4d75-92d3-d1aab2feb033-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xfwgr\" (UID: \"bc25e3fb-91ba-4d75-92d3-d1aab2feb033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.446403 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.446122    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc25e3fb-91ba-4d75-92d3-d1aab2feb033-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xfwgr\" (UID: \"bc25e3fb-91ba-4d75-92d3-d1aab2feb033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.446403 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.446152    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc25e3fb-91ba-4d75-92d3-d1aab2feb033-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xfwgr\" (UID: \"bc25e3fb-91ba-4d75-92d3-d1aab2feb033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.446403 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.446195    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc25e3fb-91ba-4d75-92d3-d1aab2feb033-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xfwgr\" (UID: \"bc25e3fb-91ba-4d75-92d3-d1aab2feb033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.446403 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.446258    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc25e3fb-91ba-4d75-92d3-d1aab2feb033-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xfwgr\" (UID: \"bc25e3fb-91ba-4d75-92d3-d1aab2feb033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.446403 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.446288    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc25e3fb-91ba-4d75-92d3-d1aab2feb033-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xfwgr\" (UID: \"bc25e3fb-91ba-4d75-92d3-d1aab2feb033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.546830 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.546799    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqkdk\" (UniqueName: \"kubernetes.io/projected/bc25e3fb-91ba-4d75-92d3-d1aab2feb033-kube-api-access-vqkdk\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xfwgr\" (UID: \"bc25e3fb-91ba-4d75-92d3-d1aab2feb033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.547018 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.546843    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc25e3fb-91ba-4d75-92d3-d1aab2feb033-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xfwgr\" (UID: \"bc25e3fb-91ba-4d75-92d3-d1aab2feb033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.547018 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.546968    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc25e3fb-91ba-4d75-92d3-d1aab2feb033-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xfwgr\" (UID: \"bc25e3fb-91ba-4d75-92d3-d1aab2feb033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.547018 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.547007    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc25e3fb-91ba-4d75-92d3-d1aab2feb033-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xfwgr\" (UID: \"bc25e3fb-91ba-4d75-92d3-d1aab2feb033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.547205 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.547034    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc25e3fb-91ba-4d75-92d3-d1aab2feb033-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xfwgr\" (UID: \"bc25e3fb-91ba-4d75-92d3-d1aab2feb033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.547205 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.547057    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc25e3fb-91ba-4d75-92d3-d1aab2feb033-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xfwgr\" (UID: \"bc25e3fb-91ba-4d75-92d3-d1aab2feb033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.547205 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.547100    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc25e3fb-91ba-4d75-92d3-d1aab2feb033-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xfwgr\" (UID: \"bc25e3fb-91ba-4d75-92d3-d1aab2feb033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.547205 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.547153    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc25e3fb-91ba-4d75-92d3-d1aab2feb033-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xfwgr\" (UID: \"bc25e3fb-91ba-4d75-92d3-d1aab2feb033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.547205 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.547193    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc25e3fb-91ba-4d75-92d3-d1aab2feb033-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xfwgr\" (UID: \"bc25e3fb-91ba-4d75-92d3-d1aab2feb033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.547482 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.547408    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc25e3fb-91ba-4d75-92d3-d1aab2feb033-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xfwgr\" (UID: \"bc25e3fb-91ba-4d75-92d3-d1aab2feb033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.547642 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.547619    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc25e3fb-91ba-4d75-92d3-d1aab2feb033-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xfwgr\" (UID: \"bc25e3fb-91ba-4d75-92d3-d1aab2feb033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.547642 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.547630    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc25e3fb-91ba-4d75-92d3-d1aab2feb033-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xfwgr\" (UID: \"bc25e3fb-91ba-4d75-92d3-d1aab2feb033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.547762 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.547652    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc25e3fb-91ba-4d75-92d3-d1aab2feb033-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xfwgr\" (UID: \"bc25e3fb-91ba-4d75-92d3-d1aab2feb033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.547762 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.547746    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc25e3fb-91ba-4d75-92d3-d1aab2feb033-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xfwgr\" (UID: \"bc25e3fb-91ba-4d75-92d3-d1aab2feb033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.549441 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.549418    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc25e3fb-91ba-4d75-92d3-d1aab2feb033-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xfwgr\" (UID: \"bc25e3fb-91ba-4d75-92d3-d1aab2feb033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.549767 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.549749    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc25e3fb-91ba-4d75-92d3-d1aab2feb033-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xfwgr\" (UID: \"bc25e3fb-91ba-4d75-92d3-d1aab2feb033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.559205 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.559178    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqkdk\" (UniqueName: \"kubernetes.io/projected/bc25e3fb-91ba-4d75-92d3-d1aab2feb033-kube-api-access-vqkdk\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xfwgr\" (UID: \"bc25e3fb-91ba-4d75-92d3-d1aab2feb033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.559312 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.559297    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc25e3fb-91ba-4d75-92d3-d1aab2feb033-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xfwgr\" (UID: \"bc25e3fb-91ba-4d75-92d3-d1aab2feb033\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.679248 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.679167    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:03.958814 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.958789    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"]
Nov 03 21:07:03.961250 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:07:03.961219    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc25e3fb_91ba_4d75_92d3_d1aab2feb033.slice/crio-c07b84d24cf410cb0feb7655820b18b9cb500dad91b8e4b1955b1325bb2db58c WatchSource:0}: Error finding container c07b84d24cf410cb0feb7655820b18b9cb500dad91b8e4b1955b1325bb2db58c: Status 404 returned error can't find the container with id c07b84d24cf410cb0feb7655820b18b9cb500dad91b8e4b1955b1325bb2db58c
Nov 03 21:07:03.963286 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.963255    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:07:03.963376 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.963336    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:07:03.963445 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:03.963384    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:07:04.471364 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:04.471322    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr" event={"ID":"bc25e3fb-91ba-4d75-92d3-d1aab2feb033","Type":"ContainerStarted","Data":"cc1b4114ba2917dc8469897bb110d57c23c5874b91bfcfee2937a1654db722f4"}
Nov 03 21:07:04.471827 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:04.471378    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr" event={"ID":"bc25e3fb-91ba-4d75-92d3-d1aab2feb033","Type":"ContainerStarted","Data":"c07b84d24cf410cb0feb7655820b18b9cb500dad91b8e4b1955b1325bb2db58c"}
Nov 03 21:07:04.498983 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:04.498932    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr" podStartSLOduration=1.498918014 podStartE2EDuration="1.498918014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:04.497059289 +0000 UTC m=+3954.630187754" watchObservedRunningTime="2026-11-03 21:07:04.498918014 +0000 UTC m=+3954.632046475"
Nov 03 21:07:04.679258 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:04.679221    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:05.681342 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:05.681303    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xfwgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body=
Nov 03 21:07:05.681839 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:05.681366    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr" podUID="bc25e3fb-91ba-4d75-92d3-d1aab2feb033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused"
Nov 03 21:07:06.680144 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:06.680109    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xfwgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body=
Nov 03 21:07:06.680308 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:06.680162    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr" podUID="bc25e3fb-91ba-4d75-92d3-d1aab2feb033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused"
Nov 03 21:07:07.679561 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:07.679523    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xfwgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body=
Nov 03 21:07:07.679990 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:07.679578    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr" podUID="bc25e3fb-91ba-4d75-92d3-d1aab2feb033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused"
Nov 03 21:07:08.679812 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:08.679772    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xfwgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body=
Nov 03 21:07:08.680248 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:08.679839    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr" podUID="bc25e3fb-91ba-4d75-92d3-d1aab2feb033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused"
Nov 03 21:07:09.680303 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:09.680263    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xfwgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body=
Nov 03 21:07:09.680807 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:09.680322    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr" podUID="bc25e3fb-91ba-4d75-92d3-d1aab2feb033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused"
Nov 03 21:07:10.679819 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:10.679780    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xfwgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body=
Nov 03 21:07:10.680008 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:10.679847    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr" podUID="bc25e3fb-91ba-4d75-92d3-d1aab2feb033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused"
Nov 03 21:07:12.680204 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:12.680160    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xfwgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:07:12.680700 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:12.680232    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr" podUID="bc25e3fb-91ba-4d75-92d3-d1aab2feb033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:07:13.720763 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:13.720726    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xfwgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:07:13.721080 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:13.720788    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr" podUID="bc25e3fb-91ba-4d75-92d3-d1aab2feb033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:07:14.762841 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:14.762800    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xfwgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:07:14.763407 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:14.762872    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr" podUID="bc25e3fb-91ba-4d75-92d3-d1aab2feb033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:07:14.985156 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:14.985127    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:14.985440 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:14.985419    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:14.986190 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:14.986171    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr"
Nov 03 21:07:18.363147 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.363112    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"]
Nov 03 21:07:18.366571 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.366551    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.381162 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.381135    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"]
Nov 03 21:07:18.503512 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.503472    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9375775d-7b1d-48d0-ab7b-ce8cf03729e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bnt6k\" (UID: \"9375775d-7b1d-48d0-ab7b-ce8cf03729e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.503512 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.503513    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9375775d-7b1d-48d0-ab7b-ce8cf03729e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bnt6k\" (UID: \"9375775d-7b1d-48d0-ab7b-ce8cf03729e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.503786 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.503538    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9375775d-7b1d-48d0-ab7b-ce8cf03729e9-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bnt6k\" (UID: \"9375775d-7b1d-48d0-ab7b-ce8cf03729e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.503786 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.503588    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhgvc\" (UniqueName: \"kubernetes.io/projected/9375775d-7b1d-48d0-ab7b-ce8cf03729e9-kube-api-access-lhgvc\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bnt6k\" (UID: \"9375775d-7b1d-48d0-ab7b-ce8cf03729e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.503786 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.503764    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9375775d-7b1d-48d0-ab7b-ce8cf03729e9-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bnt6k\" (UID: \"9375775d-7b1d-48d0-ab7b-ce8cf03729e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.503941 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.503796    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9375775d-7b1d-48d0-ab7b-ce8cf03729e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bnt6k\" (UID: \"9375775d-7b1d-48d0-ab7b-ce8cf03729e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.503941 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.503824    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9375775d-7b1d-48d0-ab7b-ce8cf03729e9-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bnt6k\" (UID: \"9375775d-7b1d-48d0-ab7b-ce8cf03729e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.503941 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.503845    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9375775d-7b1d-48d0-ab7b-ce8cf03729e9-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bnt6k\" (UID: \"9375775d-7b1d-48d0-ab7b-ce8cf03729e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.503941 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.503924    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9375775d-7b1d-48d0-ab7b-ce8cf03729e9-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bnt6k\" (UID: \"9375775d-7b1d-48d0-ab7b-ce8cf03729e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.605287 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.605253    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9375775d-7b1d-48d0-ab7b-ce8cf03729e9-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bnt6k\" (UID: \"9375775d-7b1d-48d0-ab7b-ce8cf03729e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.605287 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.605290    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9375775d-7b1d-48d0-ab7b-ce8cf03729e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bnt6k\" (UID: \"9375775d-7b1d-48d0-ab7b-ce8cf03729e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.605509 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.605313    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9375775d-7b1d-48d0-ab7b-ce8cf03729e9-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bnt6k\" (UID: \"9375775d-7b1d-48d0-ab7b-ce8cf03729e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.605509 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.605364    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9375775d-7b1d-48d0-ab7b-ce8cf03729e9-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bnt6k\" (UID: \"9375775d-7b1d-48d0-ab7b-ce8cf03729e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.605634 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.605542    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9375775d-7b1d-48d0-ab7b-ce8cf03729e9-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bnt6k\" (UID: \"9375775d-7b1d-48d0-ab7b-ce8cf03729e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.605698 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.605643    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9375775d-7b1d-48d0-ab7b-ce8cf03729e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bnt6k\" (UID: \"9375775d-7b1d-48d0-ab7b-ce8cf03729e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.605698 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.605673    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9375775d-7b1d-48d0-ab7b-ce8cf03729e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bnt6k\" (UID: \"9375775d-7b1d-48d0-ab7b-ce8cf03729e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.605830 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.605704    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9375775d-7b1d-48d0-ab7b-ce8cf03729e9-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bnt6k\" (UID: \"9375775d-7b1d-48d0-ab7b-ce8cf03729e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.605830 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.605730    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhgvc\" (UniqueName: \"kubernetes.io/projected/9375775d-7b1d-48d0-ab7b-ce8cf03729e9-kube-api-access-lhgvc\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bnt6k\" (UID: \"9375775d-7b1d-48d0-ab7b-ce8cf03729e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.605830 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.605740    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9375775d-7b1d-48d0-ab7b-ce8cf03729e9-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bnt6k\" (UID: \"9375775d-7b1d-48d0-ab7b-ce8cf03729e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.605830 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.605773    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9375775d-7b1d-48d0-ab7b-ce8cf03729e9-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bnt6k\" (UID: \"9375775d-7b1d-48d0-ab7b-ce8cf03729e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.606051 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.605920    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9375775d-7b1d-48d0-ab7b-ce8cf03729e9-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bnt6k\" (UID: \"9375775d-7b1d-48d0-ab7b-ce8cf03729e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.606110 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.606061    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9375775d-7b1d-48d0-ab7b-ce8cf03729e9-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bnt6k\" (UID: \"9375775d-7b1d-48d0-ab7b-ce8cf03729e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.606264 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.606245    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9375775d-7b1d-48d0-ab7b-ce8cf03729e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bnt6k\" (UID: \"9375775d-7b1d-48d0-ab7b-ce8cf03729e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.607993 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.607974    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9375775d-7b1d-48d0-ab7b-ce8cf03729e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bnt6k\" (UID: \"9375775d-7b1d-48d0-ab7b-ce8cf03729e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.608106 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.608093    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9375775d-7b1d-48d0-ab7b-ce8cf03729e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bnt6k\" (UID: \"9375775d-7b1d-48d0-ab7b-ce8cf03729e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.617747 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.617690    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9375775d-7b1d-48d0-ab7b-ce8cf03729e9-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bnt6k\" (UID: \"9375775d-7b1d-48d0-ab7b-ce8cf03729e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.617894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.617877    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhgvc\" (UniqueName: \"kubernetes.io/projected/9375775d-7b1d-48d0-ab7b-ce8cf03729e9-kube-api-access-lhgvc\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bnt6k\" (UID: \"9375775d-7b1d-48d0-ab7b-ce8cf03729e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.690887 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.690844    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:18.973316 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.973291    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"]
Nov 03 21:07:18.974300 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:07:18.974280    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9375775d_7b1d_48d0_ab7b_ce8cf03729e9.slice/crio-e423f677c303e897fa9db9c9eb86b30f86361abdb83f23dee28a73ad782e9ce9 WatchSource:0}: Error finding container e423f677c303e897fa9db9c9eb86b30f86361abdb83f23dee28a73ad782e9ce9: Status 404 returned error can't find the container with id e423f677c303e897fa9db9c9eb86b30f86361abdb83f23dee28a73ad782e9ce9
Nov 03 21:07:18.976554 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.976525    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:07:18.976648 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.976586    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:07:18.976648 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:18.976637    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:07:19.552324 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:19.552269    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k" event={"ID":"9375775d-7b1d-48d0-ab7b-ce8cf03729e9","Type":"ContainerStarted","Data":"4161510ab3b3f4466168cbfc65013709f523992cfe7d69915aba833306de0d69"}
Nov 03 21:07:19.553038 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:19.552993    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k" event={"ID":"9375775d-7b1d-48d0-ab7b-ce8cf03729e9","Type":"ContainerStarted","Data":"e423f677c303e897fa9db9c9eb86b30f86361abdb83f23dee28a73ad782e9ce9"}
Nov 03 21:07:19.592805 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:19.592745    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k" podStartSLOduration=1.5927309200000002 podStartE2EDuration="1.59273092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:19.589884871 +0000 UTC m=+3969.723013332" watchObservedRunningTime="2026-11-03 21:07:19.59273092 +0000 UTC m=+3969.725859381"
Nov 03 21:07:19.628289 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:07:19.628250    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:07:19.691438 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:19.691407    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:20.692684 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:20.692646    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bnt6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body=
Nov 03 21:07:20.693078 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:20.692696    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k" podUID="9375775d-7b1d-48d0-ab7b-ce8cf03729e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused"
Nov 03 21:07:21.512039 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:21.512003    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-687b69bd9-q5zvz"]
Nov 03 21:07:21.512343 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:21.512199    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-687b69bd9-q5zvz" podUID="c66f5c12-d8d2-491d-a836-fc8776ea6261" containerName="manager" containerID="cri-o://daa2a342507e607abef7f135face8ff1206b71d56ff7c86ea393efe377823207" gracePeriod=30
Nov 03 21:07:21.691155 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:21.691121    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bnt6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body=
Nov 03 21:07:21.691334 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:21.691180    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k" podUID="9375775d-7b1d-48d0-ab7b-ce8cf03729e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused"
Nov 03 21:07:21.813811 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:21.813787    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-687b69bd9-q5zvz"
Nov 03 21:07:21.937510 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:21.937479    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c66f5c12-d8d2-491d-a836-fc8776ea6261-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c66f5c12-d8d2-491d-a836-fc8776ea6261-metrics-certs\") pod \"c66f5c12-d8d2-491d-a836-fc8776ea6261\" (UID: \"c66f5c12-d8d2-491d-a836-fc8776ea6261\") "
Nov 03 21:07:21.937729 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:21.937536    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c66f5c12-d8d2-491d-a836-fc8776ea6261-webhook-cert\" (UniqueName: \"kubernetes.io/secret/c66f5c12-d8d2-491d-a836-fc8776ea6261-webhook-cert\") pod \"c66f5c12-d8d2-491d-a836-fc8776ea6261\" (UID: \"c66f5c12-d8d2-491d-a836-fc8776ea6261\") "
Nov 03 21:07:21.937729 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:21.937635    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c66f5c12-d8d2-491d-a836-fc8776ea6261-kube-api-access-ggb4b\" (UniqueName: \"kubernetes.io/projected/c66f5c12-d8d2-491d-a836-fc8776ea6261-kube-api-access-ggb4b\") pod \"c66f5c12-d8d2-491d-a836-fc8776ea6261\" (UID: \"c66f5c12-d8d2-491d-a836-fc8776ea6261\") "
Nov 03 21:07:21.940004 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:21.939971    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c66f5c12-d8d2-491d-a836-fc8776ea6261-webhook-cert" pod "c66f5c12-d8d2-491d-a836-fc8776ea6261" (UID: "c66f5c12-d8d2-491d-a836-fc8776ea6261"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:07:21.940150 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:21.940038    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c66f5c12-d8d2-491d-a836-fc8776ea6261-metrics-certs" pod "c66f5c12-d8d2-491d-a836-fc8776ea6261" (UID: "c66f5c12-d8d2-491d-a836-fc8776ea6261"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:07:21.940291 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:21.940268    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c66f5c12-d8d2-491d-a836-fc8776ea6261-kube-api-access-ggb4b" pod "c66f5c12-d8d2-491d-a836-fc8776ea6261" (UID: "c66f5c12-d8d2-491d-a836-fc8776ea6261"). InnerVolumeSpecName "kube-api-access-ggb4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:07:22.038750 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:22.038719    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c66f5c12-d8d2-491d-a836-fc8776ea6261-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:22.038919 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:22.038747    4421 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c66f5c12-d8d2-491d-a836-fc8776ea6261-webhook-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:22.038919 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:22.038771    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggb4b\" (UniqueName: \"kubernetes.io/projected/c66f5c12-d8d2-491d-a836-fc8776ea6261-kube-api-access-ggb4b\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:07:22.576899 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:22.576866    4421 generic.go:356] "Generic (PLEG): container finished" podID="c66f5c12-d8d2-491d-a836-fc8776ea6261" containerID="daa2a342507e607abef7f135face8ff1206b71d56ff7c86ea393efe377823207" exitCode=0
Nov 03 21:07:22.577087 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:22.576912    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-687b69bd9-q5zvz"
Nov 03 21:07:22.577087 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:22.576946    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-687b69bd9-q5zvz" event={"ID":"c66f5c12-d8d2-491d-a836-fc8776ea6261","Type":"ContainerDied","Data":"daa2a342507e607abef7f135face8ff1206b71d56ff7c86ea393efe377823207"}
Nov 03 21:07:22.577087 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:22.577006    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-687b69bd9-q5zvz" event={"ID":"c66f5c12-d8d2-491d-a836-fc8776ea6261","Type":"ContainerDied","Data":"6119e413f344694e0ff43027a6529f78e96cb830b4433a5e3b52770cbf1fb422"}
Nov 03 21:07:22.577087 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:22.577035    4421 scope.go:122] "RemoveContainer" containerID="daa2a342507e607abef7f135face8ff1206b71d56ff7c86ea393efe377823207"
Nov 03 21:07:22.585691 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:22.585679    4421 scope.go:122] "RemoveContainer" containerID="daa2a342507e607abef7f135face8ff1206b71d56ff7c86ea393efe377823207"
Nov 03 21:07:22.585968 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:07:22.585951    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daa2a342507e607abef7f135face8ff1206b71d56ff7c86ea393efe377823207\": container with ID starting with daa2a342507e607abef7f135face8ff1206b71d56ff7c86ea393efe377823207 not found: ID does not exist" containerID="daa2a342507e607abef7f135face8ff1206b71d56ff7c86ea393efe377823207"
Nov 03 21:07:22.586027 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:22.585975    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daa2a342507e607abef7f135face8ff1206b71d56ff7c86ea393efe377823207"} err="failed to get container status \"daa2a342507e607abef7f135face8ff1206b71d56ff7c86ea393efe377823207\": rpc error: code = NotFound desc = could not find container \"daa2a342507e607abef7f135face8ff1206b71d56ff7c86ea393efe377823207\": container with ID starting with daa2a342507e607abef7f135face8ff1206b71d56ff7c86ea393efe377823207 not found: ID does not exist"
Nov 03 21:07:22.594982 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:22.594962    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-687b69bd9-q5zvz"]
Nov 03 21:07:22.601348 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:22.601325    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-687b69bd9-q5zvz"]
Nov 03 21:07:22.691799 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:22.691767    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bnt6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body=
Nov 03 21:07:22.691956 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:22.691816    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k" podUID="9375775d-7b1d-48d0-ab7b-ce8cf03729e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused"
Nov 03 21:07:23.691610 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:23.691568    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bnt6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body=
Nov 03 21:07:23.692036 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:23.691650    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k" podUID="9375775d-7b1d-48d0-ab7b-ce8cf03729e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused"
Nov 03 21:07:23.800535 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:23.800499    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-687b69bd9-hmtjv"]
Nov 03 21:07:23.800981 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:23.800966    4421 cpu_manager.go:414] "containerMap: removing container" podUID="c66f5c12-d8d2-491d-a836-fc8776ea6261" containerName="manager"
Nov 03 21:07:23.801034 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:23.800983    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c66f5c12-d8d2-491d-a836-fc8776ea6261" containerName="manager"
Nov 03 21:07:23.801071 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:23.801059    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="c66f5c12-d8d2-491d-a836-fc8776ea6261" containerName="manager"
Nov 03 21:07:23.805801 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:23.805782    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-687b69bd9-hmtjv"
Nov 03 21:07:23.812924 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:23.812903    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 21:07:23.814086 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:23.814059    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 21:07:23.814086 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:23.814093    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 21:07:23.814371 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:23.814354    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7tsr7\""
Nov 03 21:07:23.815118 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:23.815099    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-687b69bd9-hmtjv"]
Nov 03 21:07:23.979314 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:23.979217    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x6qh\" (UniqueName: \"kubernetes.io/projected/670212d6-aae1-4981-aaf1-5b12df51bc0d-kube-api-access-9x6qh\") pod \"maas-controller-687b69bd9-hmtjv\" (UID: \"670212d6-aae1-4981-aaf1-5b12df51bc0d\") " pod="opendatahub/maas-controller-687b69bd9-hmtjv"
Nov 03 21:07:23.979314 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:23.979279    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/670212d6-aae1-4981-aaf1-5b12df51bc0d-metrics-certs\") pod \"maas-controller-687b69bd9-hmtjv\" (UID: \"670212d6-aae1-4981-aaf1-5b12df51bc0d\") " pod="opendatahub/maas-controller-687b69bd9-hmtjv"
Nov 03 21:07:23.979507 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:23.979385    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/670212d6-aae1-4981-aaf1-5b12df51bc0d-webhook-cert\") pod \"maas-controller-687b69bd9-hmtjv\" (UID: \"670212d6-aae1-4981-aaf1-5b12df51bc0d\") " pod="opendatahub/maas-controller-687b69bd9-hmtjv"
Nov 03 21:07:24.014238 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:24.014206    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c66f5c12-d8d2-491d-a836-fc8776ea6261" path="/var/lib/kubelet/pods/c66f5c12-d8d2-491d-a836-fc8776ea6261/volumes"
Nov 03 21:07:24.080650 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:24.080580    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/670212d6-aae1-4981-aaf1-5b12df51bc0d-metrics-certs\") pod \"maas-controller-687b69bd9-hmtjv\" (UID: \"670212d6-aae1-4981-aaf1-5b12df51bc0d\") " pod="opendatahub/maas-controller-687b69bd9-hmtjv"
Nov 03 21:07:24.080872 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:24.080666    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/670212d6-aae1-4981-aaf1-5b12df51bc0d-webhook-cert\") pod \"maas-controller-687b69bd9-hmtjv\" (UID: \"670212d6-aae1-4981-aaf1-5b12df51bc0d\") " pod="opendatahub/maas-controller-687b69bd9-hmtjv"
Nov 03 21:07:24.080872 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:24.080756    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x6qh\" (UniqueName: \"kubernetes.io/projected/670212d6-aae1-4981-aaf1-5b12df51bc0d-kube-api-access-9x6qh\") pod \"maas-controller-687b69bd9-hmtjv\" (UID: \"670212d6-aae1-4981-aaf1-5b12df51bc0d\") " pod="opendatahub/maas-controller-687b69bd9-hmtjv"
Nov 03 21:07:24.083380 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:24.083357    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/670212d6-aae1-4981-aaf1-5b12df51bc0d-webhook-cert\") pod \"maas-controller-687b69bd9-hmtjv\" (UID: \"670212d6-aae1-4981-aaf1-5b12df51bc0d\") " pod="opendatahub/maas-controller-687b69bd9-hmtjv"
Nov 03 21:07:24.083514 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:24.083357    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/670212d6-aae1-4981-aaf1-5b12df51bc0d-metrics-certs\") pod \"maas-controller-687b69bd9-hmtjv\" (UID: \"670212d6-aae1-4981-aaf1-5b12df51bc0d\") " pod="opendatahub/maas-controller-687b69bd9-hmtjv"
Nov 03 21:07:24.093016 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:24.092993    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x6qh\" (UniqueName: \"kubernetes.io/projected/670212d6-aae1-4981-aaf1-5b12df51bc0d-kube-api-access-9x6qh\") pod \"maas-controller-687b69bd9-hmtjv\" (UID: \"670212d6-aae1-4981-aaf1-5b12df51bc0d\") " pod="opendatahub/maas-controller-687b69bd9-hmtjv"
Nov 03 21:07:24.119606 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:24.119577    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-687b69bd9-hmtjv"
Nov 03 21:07:24.422128 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:24.422046    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-687b69bd9-hmtjv"]
Nov 03 21:07:24.425279 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:07:24.425253    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod670212d6_aae1_4981_aaf1_5b12df51bc0d.slice/crio-64dfe5b3f724e3c0ef0c135e590323e2c9dc424c297e90cd491759eb8ad8978a WatchSource:0}: Error finding container 64dfe5b3f724e3c0ef0c135e590323e2c9dc424c297e90cd491759eb8ad8978a: Status 404 returned error can't find the container with id 64dfe5b3f724e3c0ef0c135e590323e2c9dc424c297e90cd491759eb8ad8978a
Nov 03 21:07:24.588818 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:24.588780    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-687b69bd9-hmtjv" event={"ID":"670212d6-aae1-4981-aaf1-5b12df51bc0d","Type":"ContainerStarted","Data":"64dfe5b3f724e3c0ef0c135e590323e2c9dc424c297e90cd491759eb8ad8978a"}
Nov 03 21:07:24.691845 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:24.691813    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bnt6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body=
Nov 03 21:07:24.692263 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:24.691871    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k" podUID="9375775d-7b1d-48d0-ab7b-ce8cf03729e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused"
Nov 03 21:07:25.595618 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:25.595568    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-687b69bd9-hmtjv" event={"ID":"670212d6-aae1-4981-aaf1-5b12df51bc0d","Type":"ContainerStarted","Data":"5b6435acaaacb31eaeeb0c747621c259ddb0ca2222d0c2e56f2d6fa6ac926915"}
Nov 03 21:07:25.595812 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:25.595663    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-687b69bd9-hmtjv"
Nov 03 21:07:25.616741 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:25.616686    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-687b69bd9-hmtjv" podStartSLOduration=2.2000010899999998 podStartE2EDuration="2.616673327s" totalImagesPullingTime="416.672237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:25.615940836 +0000 UTC m=+3975.749069302" watchObservedRunningTime="2026-11-03 21:07:25.616673327 +0000 UTC m=+3975.749801788"
Nov 03 21:07:25.691142 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:25.691114    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bnt6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body=
Nov 03 21:07:25.691299 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:25.691167    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k" podUID="9375775d-7b1d-48d0-ab7b-ce8cf03729e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused"
Nov 03 21:07:27.555685 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:27.555653    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:27.556277 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:27.556253    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:27.557086 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:27.557064    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k"
Nov 03 21:07:36.603685 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:07:36.603650    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-687b69bd9-hmtjv"
Nov 03 21:07:49.628749 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:07:49.628660    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:08:12.498626 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:08:12.496736    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-6bhmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:08:12.498626 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:08:12.496872    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr" podUID="9d6e2ee7-7d48-4879-b6fa-f303b5d24697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:08:12.931682 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:08:12.930714    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-cd5ll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:08:12.931682 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:08:12.930801    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll" podUID="b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:08:19.627909 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:08:19.627865    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:08:31.056751 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:08:31.056707    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xfwgr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:08:31.057391 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:08:31.057356    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr" podUID="bc25e3fb-91ba-4d75-92d3-d1aab2feb033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:08:49.628687 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:08:49.628643    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:09:15.722571 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:15.722536    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv"]
Nov 03 21:09:15.730274 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:15.730244    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv"
Nov 03 21:09:15.737029 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:15.737003    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-0704dddb-epp-sa-dockercfg-82pmh\""
Nov 03 21:09:15.737662 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:15.737620    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-0704dddb-kserve-self-signed-certs\""
Nov 03 21:09:15.740178 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:15.740159    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv"]
Nov 03 21:09:15.779268 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:15.779227    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98297648-1ef2-4ad2-802b-892fa4a4a66f-tls-certs\") pod \"e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv\" (UID: \"98297648-1ef2-4ad2-802b-892fa4a4a66f\") " pod="llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv"
Nov 03 21:09:15.779454 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:15.779430    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj4gr\" (UniqueName: \"kubernetes.io/projected/98297648-1ef2-4ad2-802b-892fa4a4a66f-kube-api-access-gj4gr\") pod \"e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv\" (UID: \"98297648-1ef2-4ad2-802b-892fa4a4a66f\") " pod="llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv"
Nov 03 21:09:15.880308 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:15.880278    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj4gr\" (UniqueName: \"kubernetes.io/projected/98297648-1ef2-4ad2-802b-892fa4a4a66f-kube-api-access-gj4gr\") pod \"e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv\" (UID: \"98297648-1ef2-4ad2-802b-892fa4a4a66f\") " pod="llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv"
Nov 03 21:09:15.880515 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:15.880387    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98297648-1ef2-4ad2-802b-892fa4a4a66f-tls-certs\") pod \"e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv\" (UID: \"98297648-1ef2-4ad2-802b-892fa4a4a66f\") " pod="llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv"
Nov 03 21:09:15.883203 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:15.883172    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98297648-1ef2-4ad2-802b-892fa4a4a66f-tls-certs\") pod \"e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv\" (UID: \"98297648-1ef2-4ad2-802b-892fa4a4a66f\") " pod="llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv"
Nov 03 21:09:15.893850 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:15.893826    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj4gr\" (UniqueName: \"kubernetes.io/projected/98297648-1ef2-4ad2-802b-892fa4a4a66f-kube-api-access-gj4gr\") pod \"e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv\" (UID: \"98297648-1ef2-4ad2-802b-892fa4a4a66f\") " pod="llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv"
Nov 03 21:09:16.046157 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:16.046086    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv"
Nov 03 21:09:16.324815 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:16.324774    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv"]
Nov 03 21:09:16.327318 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:09:16.327287    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98297648_1ef2_4ad2_802b_892fa4a4a66f.slice/crio-76777e2545b4921f21f0cb0665985dd812306d7ad7f32ad70475e8052783dbd2 WatchSource:0}: Error finding container 76777e2545b4921f21f0cb0665985dd812306d7ad7f32ad70475e8052783dbd2: Status 404 returned error can't find the container with id 76777e2545b4921f21f0cb0665985dd812306d7ad7f32ad70475e8052783dbd2
Nov 03 21:09:16.346820 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:16.346793    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv" event={"ID":"98297648-1ef2-4ad2-802b-892fa4a4a66f","Type":"ContainerStarted","Data":"76777e2545b4921f21f0cb0665985dd812306d7ad7f32ad70475e8052783dbd2"}
Nov 03 21:09:18.362945 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:18.362850    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv" event={"ID":"98297648-1ef2-4ad2-802b-892fa4a4a66f","Type":"ContainerStarted","Data":"bed274510094ee0e501698e5ac3d10d2842055ae80abc0b11ba5f58abf6ba00b"}
Nov 03 21:09:18.363772 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:18.363748    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv"
Nov 03 21:09:18.389290 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:18.389179    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv" podStartSLOduration=1.620707626 podStartE2EDuration="3.389156524s" totalImagesPullingTime="1.768448898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:18.384070666 +0000 UTC m=+4088.517199129" watchObservedRunningTime="2026-11-03 21:09:18.389156524 +0000 UTC m=+4088.522284988"
Nov 03 21:09:19.728680 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:09:19.724874    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:09:40.500156 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:40.500118    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"]
Nov 03 21:09:40.500702 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:40.500308    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7" podUID="7b3aa674-8eed-440f-af1e-94112054a6f1" containerName="discovery" containerID="cri-o://37c5f9a08a41e0d5185a9ff474e35d35f61f87c28b131008a51363e2e3dd64c2" gracePeriod=30
Nov 03 21:09:40.765494 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:40.765471    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"
Nov 03 21:09:40.884689 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:40.884651    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-csr-ca-configmap\") pod \"7b3aa674-8eed-440f-af1e-94112054a6f1\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") "
Nov 03 21:09:40.884689 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:40.884692    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-kubeconfig\") pod \"7b3aa674-8eed-440f-af1e-94112054a6f1\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") "
Nov 03 21:09:40.885091 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:40.884777    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b3aa674-8eed-440f-af1e-94112054a6f1-kube-api-access-dlzcg\" (UniqueName: \"kubernetes.io/projected/7b3aa674-8eed-440f-af1e-94112054a6f1-kube-api-access-dlzcg\") pod \"7b3aa674-8eed-440f-af1e-94112054a6f1\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") "
Nov 03 21:09:40.885091 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:40.884844    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-csr-dns-cert\") pod \"7b3aa674-8eed-440f-af1e-94112054a6f1\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") "
Nov 03 21:09:40.885091 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:40.884895    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b3aa674-8eed-440f-af1e-94112054a6f1-cacerts\" (UniqueName: \"kubernetes.io/secret/7b3aa674-8eed-440f-af1e-94112054a6f1-cacerts\") pod \"7b3aa674-8eed-440f-af1e-94112054a6f1\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") "
Nov 03 21:09:40.885091 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:40.884936    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-token\" (UniqueName: \"kubernetes.io/projected/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-token\") pod \"7b3aa674-8eed-440f-af1e-94112054a6f1\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") "
Nov 03 21:09:40.885091 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:40.884972    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b3aa674-8eed-440f-af1e-94112054a6f1-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b3aa674-8eed-440f-af1e-94112054a6f1-local-certs\") pod \"7b3aa674-8eed-440f-af1e-94112054a6f1\" (UID: \"7b3aa674-8eed-440f-af1e-94112054a6f1\") "
Nov 03 21:09:40.885091 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:40.885057    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-csr-ca-configmap" pod "7b3aa674-8eed-440f-af1e-94112054a6f1" (UID: "7b3aa674-8eed-440f-af1e-94112054a6f1"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:09:40.885503 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:40.885341    4421 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-csr-ca-configmap\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:40.887743 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:40.887698    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b3aa674-8eed-440f-af1e-94112054a6f1-cacerts" pod "7b3aa674-8eed-440f-af1e-94112054a6f1" (UID: "7b3aa674-8eed-440f-af1e-94112054a6f1"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:09:40.887900 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:40.887769    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-kubeconfig" pod "7b3aa674-8eed-440f-af1e-94112054a6f1" (UID: "7b3aa674-8eed-440f-af1e-94112054a6f1"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:09:40.888050 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:40.888024    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b3aa674-8eed-440f-af1e-94112054a6f1-kube-api-access-dlzcg" pod "7b3aa674-8eed-440f-af1e-94112054a6f1" (UID: "7b3aa674-8eed-440f-af1e-94112054a6f1"). InnerVolumeSpecName "kube-api-access-dlzcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:09:40.888829 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:40.888801    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3aa674-8eed-440f-af1e-94112054a6f1-local-certs" pod "7b3aa674-8eed-440f-af1e-94112054a6f1" (UID: "7b3aa674-8eed-440f-af1e-94112054a6f1"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:09:40.888950 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:40.888897    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-csr-dns-cert" pod "7b3aa674-8eed-440f-af1e-94112054a6f1" (UID: "7b3aa674-8eed-440f-af1e-94112054a6f1"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:09:40.888950 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:40.888915    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-token" pod "7b3aa674-8eed-440f-af1e-94112054a6f1" (UID: "7b3aa674-8eed-440f-af1e-94112054a6f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:09:40.986925 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:40.986891    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlzcg\" (UniqueName: \"kubernetes.io/projected/7b3aa674-8eed-440f-af1e-94112054a6f1-kube-api-access-dlzcg\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:40.986925 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:40.986925    4421 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-csr-dns-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:40.987133 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:40.986938    4421 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7b3aa674-8eed-440f-af1e-94112054a6f1-cacerts\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:40.987133 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:40.986951    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:40.987133 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:40.986965    4421 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b3aa674-8eed-440f-af1e-94112054a6f1-local-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:40.987133 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:40.986975    4421 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b3aa674-8eed-440f-af1e-94112054a6f1-istio-kubeconfig\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:09:41.525960 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:41.525923    4421 generic.go:356] "Generic (PLEG): container finished" podID="7b3aa674-8eed-440f-af1e-94112054a6f1" containerID="37c5f9a08a41e0d5185a9ff474e35d35f61f87c28b131008a51363e2e3dd64c2" exitCode=0
Nov 03 21:09:41.526458 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:41.525992    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"
Nov 03 21:09:41.526458 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:41.526008    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7" event={"ID":"7b3aa674-8eed-440f-af1e-94112054a6f1","Type":"ContainerDied","Data":"37c5f9a08a41e0d5185a9ff474e35d35f61f87c28b131008a51363e2e3dd64c2"}
Nov 03 21:09:41.526458 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:41.526095    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7" event={"ID":"7b3aa674-8eed-440f-af1e-94112054a6f1","Type":"ContainerDied","Data":"c8cde489768aa74d93bbafb9d79c79fdc910f84149388aff71ec3ece9b3fc2a8"}
Nov 03 21:09:41.526458 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:41.526120    4421 scope.go:122] "RemoveContainer" containerID="37c5f9a08a41e0d5185a9ff474e35d35f61f87c28b131008a51363e2e3dd64c2"
Nov 03 21:09:41.554304 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:41.554276    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"]
Nov 03 21:09:41.561740 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:41.561707    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9pwj7"]
Nov 03 21:09:41.578934 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:41.578902    4421 scope.go:122] "RemoveContainer" containerID="37c5f9a08a41e0d5185a9ff474e35d35f61f87c28b131008a51363e2e3dd64c2"
Nov 03 21:09:41.579306 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:09:41.579284    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37c5f9a08a41e0d5185a9ff474e35d35f61f87c28b131008a51363e2e3dd64c2\": container with ID starting with 37c5f9a08a41e0d5185a9ff474e35d35f61f87c28b131008a51363e2e3dd64c2 not found: ID does not exist" containerID="37c5f9a08a41e0d5185a9ff474e35d35f61f87c28b131008a51363e2e3dd64c2"
Nov 03 21:09:41.579380 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:41.579317    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37c5f9a08a41e0d5185a9ff474e35d35f61f87c28b131008a51363e2e3dd64c2"} err="failed to get container status \"37c5f9a08a41e0d5185a9ff474e35d35f61f87c28b131008a51363e2e3dd64c2\": rpc error: code = NotFound desc = could not find container \"37c5f9a08a41e0d5185a9ff474e35d35f61f87c28b131008a51363e2e3dd64c2\": container with ID starting with 37c5f9a08a41e0d5185a9ff474e35d35f61f87c28b131008a51363e2e3dd64c2 not found: ID does not exist"
Nov 03 21:09:42.014512 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:42.014478    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b3aa674-8eed-440f-af1e-94112054a6f1" path="/var/lib/kubelet/pods/7b3aa674-8eed-440f-af1e-94112054a6f1/volumes"
Nov 03 21:09:49.627767 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:09:49.627727    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:09:50.416310 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:09:50.416283    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv"
Nov 03 21:10:15.881507 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:10:15.881472    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv"]
Nov 03 21:10:19.733839 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:10:19.733766    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:10:20.420662 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:10:20.420615    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv" podUID="98297648-1ef2-4ad2-802b-892fa4a4a66f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:10:30.416838 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:10:30.416800    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv" podUID="98297648-1ef2-4ad2-802b-892fa4a4a66f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:10:31.251017 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:10:31.250974    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv" podUID="98297648-1ef2-4ad2-802b-892fa4a4a66f" containerName="main" containerID="cri-o://bed274510094ee0e501698e5ac3d10d2842055ae80abc0b11ba5f58abf6ba00b" gracePeriod=45
Nov 03 21:10:40.416675 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:10:40.416631    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv" podUID="98297648-1ef2-4ad2-802b-892fa4a4a66f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:10:40.417095 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:10:40.416748    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv"
Nov 03 21:10:49.628068 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:10:49.627984    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:10:50.416455 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:10:50.416413    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv" podUID="98297648-1ef2-4ad2-802b-892fa4a4a66f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:11:00.416089 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:11:00.416043    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv" podUID="98297648-1ef2-4ad2-802b-892fa4a4a66f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:11:01.509142 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:11:01.509070    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv"
Nov 03 21:11:01.586706 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:11:01.586677    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98297648-1ef2-4ad2-802b-892fa4a4a66f-tls-certs\" (UniqueName: \"kubernetes.io/secret/98297648-1ef2-4ad2-802b-892fa4a4a66f-tls-certs\") pod \"98297648-1ef2-4ad2-802b-892fa4a4a66f\" (UID: \"98297648-1ef2-4ad2-802b-892fa4a4a66f\") "
Nov 03 21:11:01.586873 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:11:01.586840    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98297648-1ef2-4ad2-802b-892fa4a4a66f-kube-api-access-gj4gr\" (UniqueName: \"kubernetes.io/projected/98297648-1ef2-4ad2-802b-892fa4a4a66f-kube-api-access-gj4gr\") pod \"98297648-1ef2-4ad2-802b-892fa4a4a66f\" (UID: \"98297648-1ef2-4ad2-802b-892fa4a4a66f\") "
Nov 03 21:11:01.589063 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:11:01.589035    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98297648-1ef2-4ad2-802b-892fa4a4a66f-tls-certs" pod "98297648-1ef2-4ad2-802b-892fa4a4a66f" (UID: "98297648-1ef2-4ad2-802b-892fa4a4a66f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:11:01.589125 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:11:01.589097    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98297648-1ef2-4ad2-802b-892fa4a4a66f-kube-api-access-gj4gr" pod "98297648-1ef2-4ad2-802b-892fa4a4a66f" (UID: "98297648-1ef2-4ad2-802b-892fa4a4a66f"). InnerVolumeSpecName "kube-api-access-gj4gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:11:01.688336 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:11:01.688309    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj4gr\" (UniqueName: \"kubernetes.io/projected/98297648-1ef2-4ad2-802b-892fa4a4a66f-kube-api-access-gj4gr\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:01.688336 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:11:01.688333    4421 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98297648-1ef2-4ad2-802b-892fa4a4a66f-tls-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:11:01.940135 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:11:01.940106    4421 generic.go:356] "Generic (PLEG): container finished" podID="98297648-1ef2-4ad2-802b-892fa4a4a66f" containerID="bed274510094ee0e501698e5ac3d10d2842055ae80abc0b11ba5f58abf6ba00b" exitCode=0
Nov 03 21:11:01.940317 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:11:01.940169    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv" event={"ID":"98297648-1ef2-4ad2-802b-892fa4a4a66f","Type":"ContainerDied","Data":"bed274510094ee0e501698e5ac3d10d2842055ae80abc0b11ba5f58abf6ba00b"}
Nov 03 21:11:01.940317 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:11:01.940212    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv" event={"ID":"98297648-1ef2-4ad2-802b-892fa4a4a66f","Type":"ContainerDied","Data":"76777e2545b4921f21f0cb0665985dd812306d7ad7f32ad70475e8052783dbd2"}
Nov 03 21:11:01.940317 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:11:01.940223    4421 scope.go:122] "RemoveContainer" containerID="bed274510094ee0e501698e5ac3d10d2842055ae80abc0b11ba5f58abf6ba00b"
Nov 03 21:11:01.940317 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:11:01.940212    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv"
Nov 03 21:11:01.951066 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:11:01.951041    4421 scope.go:122] "RemoveContainer" containerID="bed274510094ee0e501698e5ac3d10d2842055ae80abc0b11ba5f58abf6ba00b"
Nov 03 21:11:01.951504 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:11:01.951434    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bed274510094ee0e501698e5ac3d10d2842055ae80abc0b11ba5f58abf6ba00b\": container with ID starting with bed274510094ee0e501698e5ac3d10d2842055ae80abc0b11ba5f58abf6ba00b not found: ID does not exist" containerID="bed274510094ee0e501698e5ac3d10d2842055ae80abc0b11ba5f58abf6ba00b"
Nov 03 21:11:01.951504 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:11:01.951470    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bed274510094ee0e501698e5ac3d10d2842055ae80abc0b11ba5f58abf6ba00b"} err="failed to get container status \"bed274510094ee0e501698e5ac3d10d2842055ae80abc0b11ba5f58abf6ba00b\": rpc error: code = NotFound desc = could not find container \"bed274510094ee0e501698e5ac3d10d2842055ae80abc0b11ba5f58abf6ba00b\": container with ID starting with bed274510094ee0e501698e5ac3d10d2842055ae80abc0b11ba5f58abf6ba00b not found: ID does not exist"
Nov 03 21:11:01.962764 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:11:01.962741    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv"]
Nov 03 21:11:01.966944 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:11:01.966910    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-0704dddb-kserve-router-scheduler-58d56f94c-ft5gv"]
Nov 03 21:11:02.014897 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:11:02.014865    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98297648-1ef2-4ad2-802b-892fa4a4a66f" path="/var/lib/kubelet/pods/98297648-1ef2-4ad2-802b-892fa4a4a66f/volumes"
Nov 03 21:11:19.627690 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:11:19.627654    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:11:49.627420 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:11:49.627382    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:12:11.859758 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:11.859716    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq"]
Nov 03 21:12:11.860779 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:11.860756    4421 cpu_manager.go:414] "containerMap: removing container" podUID="98297648-1ef2-4ad2-802b-892fa4a4a66f" containerName="main"
Nov 03 21:12:11.860779 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:11.860778    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98297648-1ef2-4ad2-802b-892fa4a4a66f" containerName="main"
Nov 03 21:12:11.860974 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:11.860811    4421 cpu_manager.go:414] "containerMap: removing container" podUID="7b3aa674-8eed-440f-af1e-94112054a6f1" containerName="discovery"
Nov 03 21:12:11.860974 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:11.860819    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b3aa674-8eed-440f-af1e-94112054a6f1" containerName="discovery"
Nov 03 21:12:11.860974 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:11.860919    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b3aa674-8eed-440f-af1e-94112054a6f1" containerName="discovery"
Nov 03 21:12:11.860974 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:11.860941    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="98297648-1ef2-4ad2-802b-892fa4a4a66f" containerName="main"
Nov 03 21:12:11.864021 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:11.864002    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq"
Nov 03 21:12:11.873314 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:11.873291    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-debeb1-serving-cert\""
Nov 03 21:12:11.877156 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:11.877057    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq"]
Nov 03 21:12:11.887056 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:11.887030    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp"]
Nov 03 21:12:11.891414 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:11.891389    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp"
Nov 03 21:12:11.895444 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:11.895421    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-debeb1\""
Nov 03 21:12:11.903150 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:11.903125    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp"]
Nov 03 21:12:11.951119 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:11.951083    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m"]
Nov 03 21:12:11.954743 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:11.954722    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m"
Nov 03 21:12:11.967087 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:11.967061    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m"]
Nov 03 21:12:11.975103 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:11.975076    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-de9dd262-dockercfg-c68lz\""
Nov 03 21:12:11.983556 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:11.983530    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpv6n\" (UniqueName: \"kubernetes.io/projected/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-kube-api-access-gpv6n\") pod \"maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq\" (UID: \"6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq"
Nov 03 21:12:11.983695 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:11.983564    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54j66\" (UniqueName: \"kubernetes.io/projected/65efecb5-5ece-42fe-9856-b69ed026f018-kube-api-access-54j66\") pod \"payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp\" (UID: \"65efecb5-5ece-42fe-9856-b69ed026f018\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp"
Nov 03 21:12:11.983695 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:11.983589    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq\" (UID: \"6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq"
Nov 03 21:12:11.983810 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:11.983783    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-metrics-certs\") pod \"maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq\" (UID: \"6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq"
Nov 03 21:12:11.983879 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:11.983866    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq\" (UID: \"6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq"
Nov 03 21:12:11.983930 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:11.983915    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65efecb5-5ece-42fe-9856-b69ed026f018-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp\" (UID: \"65efecb5-5ece-42fe-9856-b69ed026f018\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp"
Nov 03 21:12:12.085323 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:12.085286    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfp6j\" (UniqueName: \"kubernetes.io/projected/8e60bae7-9a59-4503-96db-57bc18e3660c-kube-api-access-mfp6j\") pod \"payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m\" (UID: \"8e60bae7-9a59-4503-96db-57bc18e3660c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m"
Nov 03 21:12:12.085323 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:12.085323    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-metrics-certs\") pod \"maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq\" (UID: \"6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq"
Nov 03 21:12:12.085655 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:12.085368    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq\" (UID: \"6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq"
Nov 03 21:12:12.085655 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:12.085481    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65efecb5-5ece-42fe-9856-b69ed026f018-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp\" (UID: \"65efecb5-5ece-42fe-9856-b69ed026f018\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp"
Nov 03 21:12:12.085655 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:12.085550    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e60bae7-9a59-4503-96db-57bc18e3660c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m\" (UID: \"8e60bae7-9a59-4503-96db-57bc18e3660c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m"
Nov 03 21:12:12.085655 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:12.085617    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpv6n\" (UniqueName: \"kubernetes.io/projected/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-kube-api-access-gpv6n\") pod \"maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq\" (UID: \"6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq"
Nov 03 21:12:12.085655 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:12.085646    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54j66\" (UniqueName: \"kubernetes.io/projected/65efecb5-5ece-42fe-9856-b69ed026f018-kube-api-access-54j66\") pod \"payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp\" (UID: \"65efecb5-5ece-42fe-9856-b69ed026f018\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp"
Nov 03 21:12:12.085980 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:12.085672    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq\" (UID: \"6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq"
Nov 03 21:12:12.086478 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:12.086455    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65efecb5-5ece-42fe-9856-b69ed026f018-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp\" (UID: \"65efecb5-5ece-42fe-9856-b69ed026f018\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp"
Nov 03 21:12:12.086762 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:12.086739    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq\" (UID: \"6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq"
Nov 03 21:12:12.088504 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:12.088479    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq\" (UID: \"6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq"
Nov 03 21:12:12.088964 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:12.088918    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-metrics-certs\") pod \"maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq\" (UID: \"6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq"
Nov 03 21:12:12.103639 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:12.099001    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54j66\" (UniqueName: \"kubernetes.io/projected/65efecb5-5ece-42fe-9856-b69ed026f018-kube-api-access-54j66\") pod \"payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp\" (UID: \"65efecb5-5ece-42fe-9856-b69ed026f018\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp"
Nov 03 21:12:12.103639 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:12.100765    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpv6n\" (UniqueName: \"kubernetes.io/projected/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-kube-api-access-gpv6n\") pod \"maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq\" (UID: \"6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq"
Nov 03 21:12:12.179953 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:12.179874    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq"
Nov 03 21:12:12.186242 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:12.186212    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfp6j\" (UniqueName: \"kubernetes.io/projected/8e60bae7-9a59-4503-96db-57bc18e3660c-kube-api-access-mfp6j\") pod \"payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m\" (UID: \"8e60bae7-9a59-4503-96db-57bc18e3660c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m"
Nov 03 21:12:12.186383 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:12.186364    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e60bae7-9a59-4503-96db-57bc18e3660c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m\" (UID: \"8e60bae7-9a59-4503-96db-57bc18e3660c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m"
Nov 03 21:12:12.187056 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:12.187031    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e60bae7-9a59-4503-96db-57bc18e3660c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m\" (UID: \"8e60bae7-9a59-4503-96db-57bc18e3660c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m"
Nov 03 21:12:12.199248 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:12.199223    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfp6j\" (UniqueName: \"kubernetes.io/projected/8e60bae7-9a59-4503-96db-57bc18e3660c-kube-api-access-mfp6j\") pod \"payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m\" (UID: \"8e60bae7-9a59-4503-96db-57bc18e3660c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m"
Nov 03 21:12:12.208284 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:12.208260    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp"
Nov 03 21:12:12.271427 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:12.271184    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m"
Nov 03 21:12:12.461202 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:12.461176    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq"]
Nov 03 21:12:12.464458 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:12:12.464424    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebaef21_c09d_43ee_ae6a_25fdfc4b6b99.slice/crio-b243dc599b00b98ce8a5a2b89695122fabc4f1bb236ba6931325441d68322a1e WatchSource:0}: Error finding container b243dc599b00b98ce8a5a2b89695122fabc4f1bb236ba6931325441d68322a1e: Status 404 returned error can't find the container with id b243dc599b00b98ce8a5a2b89695122fabc4f1bb236ba6931325441d68322a1e
Nov 03 21:12:12.465933 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:12.465912    4421 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:12:12.495653 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:12.495487    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp"]
Nov 03 21:12:12.497657 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:12:12.497628    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65efecb5_5ece_42fe_9856_b69ed026f018.slice/crio-354c95a123a597a166df6417373ac8cc3fb2e6fabebd10f826f57b4b68b8c81d WatchSource:0}: Error finding container 354c95a123a597a166df6417373ac8cc3fb2e6fabebd10f826f57b4b68b8c81d: Status 404 returned error can't find the container with id 354c95a123a597a166df6417373ac8cc3fb2e6fabebd10f826f57b4b68b8c81d
Nov 03 21:12:12.559706 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:12.559621    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m"]
Nov 03 21:12:12.561660 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:12:12.561636    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e60bae7_9a59_4503_96db_57bc18e3660c.slice/crio-181a05c90e616c15b622da272400951784c4e64e470fc4b8299c1039dfd46c8c WatchSource:0}: Error finding container 181a05c90e616c15b622da272400951784c4e64e470fc4b8299c1039dfd46c8c: Status 404 returned error can't find the container with id 181a05c90e616c15b622da272400951784c4e64e470fc4b8299c1039dfd46c8c
Nov 03 21:12:13.264398 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:13.264363    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m" event={"ID":"8e60bae7-9a59-4503-96db-57bc18e3660c","Type":"ContainerStarted","Data":"fa80ecb2f6ef4e4382a12bec420baa8fd69e610a69781d5c7513f862bade0879"}
Nov 03 21:12:13.264920 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:13.264420    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m" event={"ID":"8e60bae7-9a59-4503-96db-57bc18e3660c","Type":"ContainerStarted","Data":"181a05c90e616c15b622da272400951784c4e64e470fc4b8299c1039dfd46c8c"}
Nov 03 21:12:13.264920 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:13.264475    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m"
Nov 03 21:12:13.265962 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:13.265936    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq" event={"ID":"6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99","Type":"ContainerStarted","Data":"7f27a8ae9ecc51d4681d4419c7472ca7368b760167f81ac41888b074557e3682"}
Nov 03 21:12:13.266121 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:13.265975    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq" event={"ID":"6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99","Type":"ContainerStarted","Data":"b243dc599b00b98ce8a5a2b89695122fabc4f1bb236ba6931325441d68322a1e"}
Nov 03 21:12:13.266121 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:13.266003    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq"
Nov 03 21:12:13.267400 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:13.267369    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp" event={"ID":"65efecb5-5ece-42fe-9856-b69ed026f018","Type":"ContainerStarted","Data":"b71f9b6691112dfd4a7dddf12c6ecb347ef44f948eedf981af2e93ed8746f624"}
Nov 03 21:12:13.267513 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:13.267418    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp"
Nov 03 21:12:13.267513 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:13.267435    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp" event={"ID":"65efecb5-5ece-42fe-9856-b69ed026f018","Type":"ContainerStarted","Data":"354c95a123a597a166df6417373ac8cc3fb2e6fabebd10f826f57b4b68b8c81d"}
Nov 03 21:12:13.284450 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:13.284406    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m" podStartSLOduration=2.284395706 podStartE2EDuration="2.284395706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:12:13.28194501 +0000 UTC m=+4263.415073472" watchObservedRunningTime="2026-11-03 21:12:13.284395706 +0000 UTC m=+4263.417524196"
Nov 03 21:12:13.300470 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:13.300420    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp" podStartSLOduration=2.300406708 podStartE2EDuration="2.300406708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:12:13.29925048 +0000 UTC m=+4263.432378943" watchObservedRunningTime="2026-11-03 21:12:13.300406708 +0000 UTC m=+4263.433535171"
Nov 03 21:12:13.320000 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:13.319950    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq" podStartSLOduration=1.905009969 podStartE2EDuration="2.319935606s" totalImagesPullingTime="414.925637ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:12:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:12:13.318449065 +0000 UTC m=+4263.451577530" watchObservedRunningTime="2026-11-03 21:12:13.319935606 +0000 UTC m=+4263.453064067"
Nov 03 21:12:19.277899 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:19.277870    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq"
Nov 03 21:12:19.628186 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:12:19.628068    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:12:24.275017 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:24.274982    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp"
Nov 03 21:12:24.275563 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:24.275424    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m"
Nov 03 21:12:30.023689 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:30.023654    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m"]
Nov 03 21:12:30.035362 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:30.035331    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp"]
Nov 03 21:12:30.801338 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:30.801296    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq"]
Nov 03 21:12:30.801688 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:30.801630    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq" podUID="6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99" containerName="maas-api" containerID="cri-o://7f27a8ae9ecc51d4681d4419c7472ca7368b760167f81ac41888b074557e3682" gracePeriod=30
Nov 03 21:12:31.073317 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:31.073283    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq"
Nov 03 21:12:31.158272 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:31.158240    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-kube-api-access-gpv6n\" (UniqueName: \"kubernetes.io/projected/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-kube-api-access-gpv6n\") pod \"6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99\" (UID: \"6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99\") "
Nov 03 21:12:31.158481 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:31.158307    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-openshift-service-ca\") pod \"6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99\" (UID: \"6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99\") "
Nov 03 21:12:31.158481 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:31.158383    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-metrics-certs\") pod \"6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99\" (UID: \"6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99\") "
Nov 03 21:12:31.158481 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:31.158436    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-maas-api-tls\") pod \"6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99\" (UID: \"6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99\") "
Nov 03 21:12:31.158730 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:31.158703    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-openshift-service-ca" pod "6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99" (UID: "6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:12:31.160529 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:31.160502    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-metrics-certs" pod "6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99" (UID: "6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:12:31.160728 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:31.160680    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-maas-api-tls" pod "6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99" (UID: "6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:12:31.160728 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:31.160694    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-kube-api-access-gpv6n" pod "6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99" (UID: "6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99"). InnerVolumeSpecName "kube-api-access-gpv6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:12:31.259123 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:31.259092    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:31.259123 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:31.259123    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpv6n\" (UniqueName: \"kubernetes.io/projected/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-kube-api-access-gpv6n\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:31.259304 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:31.259134    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:31.259304 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:31.259143    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:31.357671 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:31.357562    4421 generic.go:356] "Generic (PLEG): container finished" podID="6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99" containerID="7f27a8ae9ecc51d4681d4419c7472ca7368b760167f81ac41888b074557e3682" exitCode=0
Nov 03 21:12:31.357671 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:31.357618    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq"
Nov 03 21:12:31.357671 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:31.357643    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq" event={"ID":"6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99","Type":"ContainerDied","Data":"7f27a8ae9ecc51d4681d4419c7472ca7368b760167f81ac41888b074557e3682"}
Nov 03 21:12:31.357976 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:31.357710    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq" event={"ID":"6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99","Type":"ContainerDied","Data":"b243dc599b00b98ce8a5a2b89695122fabc4f1bb236ba6931325441d68322a1e"}
Nov 03 21:12:31.357976 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:31.357741    4421 scope.go:122] "RemoveContainer" containerID="7f27a8ae9ecc51d4681d4419c7472ca7368b760167f81ac41888b074557e3682"
Nov 03 21:12:31.369224 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:31.369207    4421 scope.go:122] "RemoveContainer" containerID="7f27a8ae9ecc51d4681d4419c7472ca7368b760167f81ac41888b074557e3682"
Nov 03 21:12:31.369499 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:12:31.369480    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f27a8ae9ecc51d4681d4419c7472ca7368b760167f81ac41888b074557e3682\": container with ID starting with 7f27a8ae9ecc51d4681d4419c7472ca7368b760167f81ac41888b074557e3682 not found: ID does not exist" containerID="7f27a8ae9ecc51d4681d4419c7472ca7368b760167f81ac41888b074557e3682"
Nov 03 21:12:31.369555 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:31.369507    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f27a8ae9ecc51d4681d4419c7472ca7368b760167f81ac41888b074557e3682"} err="failed to get container status \"7f27a8ae9ecc51d4681d4419c7472ca7368b760167f81ac41888b074557e3682\": rpc error: code = NotFound desc = could not find container \"7f27a8ae9ecc51d4681d4419c7472ca7368b760167f81ac41888b074557e3682\": container with ID starting with 7f27a8ae9ecc51d4681d4419c7472ca7368b760167f81ac41888b074557e3682 not found: ID does not exist"
Nov 03 21:12:31.380235 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:31.380213    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq"]
Nov 03 21:12:31.384674 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:31.384654    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-f42sq"]
Nov 03 21:12:32.013662 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:32.013627    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99" path="/var/lib/kubelet/pods/6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99/volumes"
Nov 03 21:12:35.024571 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.024509    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m" podUID="8e60bae7-9a59-4503-96db-57bc18e3660c" containerName="payload-processing" containerID="cri-o://fa80ecb2f6ef4e4382a12bec420baa8fd69e610a69781d5c7513f862bade0879" gracePeriod=25
Nov 03 21:12:35.036076 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.036039    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp" podUID="65efecb5-5ece-42fe-9856-b69ed026f018" containerName="payload-pre-processing" containerID="cri-o://b71f9b6691112dfd4a7dddf12c6ecb347ef44f948eedf981af2e93ed8746f624" gracePeriod=25
Nov 03 21:12:35.291094 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.290959    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m"
Nov 03 21:12:35.305658 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.305634    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp"
Nov 03 21:12:35.378682 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.378649    4421 generic.go:356] "Generic (PLEG): container finished" podID="8e60bae7-9a59-4503-96db-57bc18e3660c" containerID="fa80ecb2f6ef4e4382a12bec420baa8fd69e610a69781d5c7513f862bade0879" exitCode=0
Nov 03 21:12:35.378872 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.378696    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m"
Nov 03 21:12:35.378872 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.378730    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m" event={"ID":"8e60bae7-9a59-4503-96db-57bc18e3660c","Type":"ContainerDied","Data":"fa80ecb2f6ef4e4382a12bec420baa8fd69e610a69781d5c7513f862bade0879"}
Nov 03 21:12:35.378872 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.378786    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m" event={"ID":"8e60bae7-9a59-4503-96db-57bc18e3660c","Type":"ContainerDied","Data":"181a05c90e616c15b622da272400951784c4e64e470fc4b8299c1039dfd46c8c"}
Nov 03 21:12:35.378872 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.378820    4421 scope.go:122] "RemoveContainer" containerID="fa80ecb2f6ef4e4382a12bec420baa8fd69e610a69781d5c7513f862bade0879"
Nov 03 21:12:35.379987 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.379966    4421 generic.go:356] "Generic (PLEG): container finished" podID="65efecb5-5ece-42fe-9856-b69ed026f018" containerID="b71f9b6691112dfd4a7dddf12c6ecb347ef44f948eedf981af2e93ed8746f624" exitCode=0
Nov 03 21:12:35.380067 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.380008    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp"
Nov 03 21:12:35.380109 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.380009    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp" event={"ID":"65efecb5-5ece-42fe-9856-b69ed026f018","Type":"ContainerDied","Data":"b71f9b6691112dfd4a7dddf12c6ecb347ef44f948eedf981af2e93ed8746f624"}
Nov 03 21:12:35.380155 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.380127    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp" event={"ID":"65efecb5-5ece-42fe-9856-b69ed026f018","Type":"ContainerDied","Data":"354c95a123a597a166df6417373ac8cc3fb2e6fabebd10f826f57b4b68b8c81d"}
Nov 03 21:12:35.389685 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.389662    4421 scope.go:122] "RemoveContainer" containerID="fa80ecb2f6ef4e4382a12bec420baa8fd69e610a69781d5c7513f862bade0879"
Nov 03 21:12:35.390046 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:12:35.390016    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa80ecb2f6ef4e4382a12bec420baa8fd69e610a69781d5c7513f862bade0879\": container with ID starting with fa80ecb2f6ef4e4382a12bec420baa8fd69e610a69781d5c7513f862bade0879 not found: ID does not exist" containerID="fa80ecb2f6ef4e4382a12bec420baa8fd69e610a69781d5c7513f862bade0879"
Nov 03 21:12:35.390150 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.390045    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa80ecb2f6ef4e4382a12bec420baa8fd69e610a69781d5c7513f862bade0879"} err="failed to get container status \"fa80ecb2f6ef4e4382a12bec420baa8fd69e610a69781d5c7513f862bade0879\": rpc error: code = NotFound desc = could not find container \"fa80ecb2f6ef4e4382a12bec420baa8fd69e610a69781d5c7513f862bade0879\": container with ID starting with fa80ecb2f6ef4e4382a12bec420baa8fd69e610a69781d5c7513f862bade0879 not found: ID does not exist"
Nov 03 21:12:35.390150 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.390066    4421 scope.go:122] "RemoveContainer" containerID="b71f9b6691112dfd4a7dddf12c6ecb347ef44f948eedf981af2e93ed8746f624"
Nov 03 21:12:35.397712 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.397692    4421 scope.go:122] "RemoveContainer" containerID="b71f9b6691112dfd4a7dddf12c6ecb347ef44f948eedf981af2e93ed8746f624"
Nov 03 21:12:35.397952 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:12:35.397931    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b71f9b6691112dfd4a7dddf12c6ecb347ef44f948eedf981af2e93ed8746f624\": container with ID starting with b71f9b6691112dfd4a7dddf12c6ecb347ef44f948eedf981af2e93ed8746f624 not found: ID does not exist" containerID="b71f9b6691112dfd4a7dddf12c6ecb347ef44f948eedf981af2e93ed8746f624"
Nov 03 21:12:35.398012 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.397962    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b71f9b6691112dfd4a7dddf12c6ecb347ef44f948eedf981af2e93ed8746f624"} err="failed to get container status \"b71f9b6691112dfd4a7dddf12c6ecb347ef44f948eedf981af2e93ed8746f624\": rpc error: code = NotFound desc = could not find container \"b71f9b6691112dfd4a7dddf12c6ecb347ef44f948eedf981af2e93ed8746f624\": container with ID starting with b71f9b6691112dfd4a7dddf12c6ecb347ef44f948eedf981af2e93ed8746f624 not found: ID does not exist"
Nov 03 21:12:35.400272 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.400255    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65efecb5-5ece-42fe-9856-b69ed026f018-kube-api-access-54j66\" (UniqueName: \"kubernetes.io/projected/65efecb5-5ece-42fe-9856-b69ed026f018-kube-api-access-54j66\") pod \"65efecb5-5ece-42fe-9856-b69ed026f018\" (UID: \"65efecb5-5ece-42fe-9856-b69ed026f018\") "
Nov 03 21:12:35.400337 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.400289    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65efecb5-5ece-42fe-9856-b69ed026f018-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65efecb5-5ece-42fe-9856-b69ed026f018-plugins-config-volume\") pod \"65efecb5-5ece-42fe-9856-b69ed026f018\" (UID: \"65efecb5-5ece-42fe-9856-b69ed026f018\") "
Nov 03 21:12:35.400337 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.400312    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e60bae7-9a59-4503-96db-57bc18e3660c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e60bae7-9a59-4503-96db-57bc18e3660c-plugins-config-volume\") pod \"8e60bae7-9a59-4503-96db-57bc18e3660c\" (UID: \"8e60bae7-9a59-4503-96db-57bc18e3660c\") "
Nov 03 21:12:35.400337 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.400330    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e60bae7-9a59-4503-96db-57bc18e3660c-kube-api-access-mfp6j\" (UniqueName: \"kubernetes.io/projected/8e60bae7-9a59-4503-96db-57bc18e3660c-kube-api-access-mfp6j\") pod \"8e60bae7-9a59-4503-96db-57bc18e3660c\" (UID: \"8e60bae7-9a59-4503-96db-57bc18e3660c\") "
Nov 03 21:12:35.400684 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.400660    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65efecb5-5ece-42fe-9856-b69ed026f018-plugins-config-volume" pod "65efecb5-5ece-42fe-9856-b69ed026f018" (UID: "65efecb5-5ece-42fe-9856-b69ed026f018"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:12:35.400782 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.400688    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e60bae7-9a59-4503-96db-57bc18e3660c-plugins-config-volume" pod "8e60bae7-9a59-4503-96db-57bc18e3660c" (UID: "8e60bae7-9a59-4503-96db-57bc18e3660c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:12:35.402347 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.402329    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65efecb5-5ece-42fe-9856-b69ed026f018-kube-api-access-54j66" pod "65efecb5-5ece-42fe-9856-b69ed026f018" (UID: "65efecb5-5ece-42fe-9856-b69ed026f018"). InnerVolumeSpecName "kube-api-access-54j66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:12:35.402470 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.402455    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e60bae7-9a59-4503-96db-57bc18e3660c-kube-api-access-mfp6j" pod "8e60bae7-9a59-4503-96db-57bc18e3660c" (UID: "8e60bae7-9a59-4503-96db-57bc18e3660c"). InnerVolumeSpecName "kube-api-access-mfp6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:12:35.501369 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.501333    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54j66\" (UniqueName: \"kubernetes.io/projected/65efecb5-5ece-42fe-9856-b69ed026f018-kube-api-access-54j66\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:35.501369 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.501363    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65efecb5-5ece-42fe-9856-b69ed026f018-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:35.501369 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.501373    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e60bae7-9a59-4503-96db-57bc18e3660c-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:35.501622 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.501382    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfp6j\" (UniqueName: \"kubernetes.io/projected/8e60bae7-9a59-4503-96db-57bc18e3660c-kube-api-access-mfp6j\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:12:35.703265 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.703239    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m"]
Nov 03 21:12:35.707345 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.707320    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fghb8m"]
Nov 03 21:12:35.718636 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.718616    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp"]
Nov 03 21:12:35.722399 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:35.722377    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694c9ldp"]
Nov 03 21:12:36.014442 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:36.014359    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65efecb5-5ece-42fe-9856-b69ed026f018" path="/var/lib/kubelet/pods/65efecb5-5ece-42fe-9856-b69ed026f018/volumes"
Nov 03 21:12:36.014797 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:36.014783    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e60bae7-9a59-4503-96db-57bc18e3660c" path="/var/lib/kubelet/pods/8e60bae7-9a59-4503-96db-57bc18e3660c/volumes"
Nov 03 21:12:36.818267 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:36.818229    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"]
Nov 03 21:12:36.818857 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:36.818811    4421 cpu_manager.go:414] "containerMap: removing container" podUID="65efecb5-5ece-42fe-9856-b69ed026f018" containerName="payload-pre-processing"
Nov 03 21:12:36.818857 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:36.818836    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65efecb5-5ece-42fe-9856-b69ed026f018" containerName="payload-pre-processing"
Nov 03 21:12:36.818857 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:36.818848    4421 cpu_manager.go:414] "containerMap: removing container" podUID="8e60bae7-9a59-4503-96db-57bc18e3660c" containerName="payload-processing"
Nov 03 21:12:36.818857 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:36.818853    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e60bae7-9a59-4503-96db-57bc18e3660c" containerName="payload-processing"
Nov 03 21:12:36.819220 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:36.818874    4421 cpu_manager.go:414] "containerMap: removing container" podUID="6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99" containerName="maas-api"
Nov 03 21:12:36.819220 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:36.818879    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99" containerName="maas-api"
Nov 03 21:12:36.819220 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:36.818974    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="65efecb5-5ece-42fe-9856-b69ed026f018" containerName="payload-pre-processing"
Nov 03 21:12:36.819220 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:36.818983    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e60bae7-9a59-4503-96db-57bc18e3660c" containerName="payload-processing"
Nov 03 21:12:36.819220 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:36.818996    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ebaef21-c09d-43ee-ae6a-25fdfc4b6b99" containerName="maas-api"
Nov 03 21:12:36.823436 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:36.823419    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"
Nov 03 21:12:36.833487 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:36.833458    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-29948937-kserve-self-signed-certs\""
Nov 03 21:12:36.835522 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:36.835497    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"]
Nov 03 21:12:36.914903 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:36.914873    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-tmp-dir\") pod \"e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv\" (UID: \"f40225b3-db40-4aa8-a465-98a998aa9549\") " pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"
Nov 03 21:12:36.915060 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:36.914924    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-dshm\") pod \"e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv\" (UID: \"f40225b3-db40-4aa8-a465-98a998aa9549\") " pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"
Nov 03 21:12:36.915060 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:36.914998    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-home\") pod \"e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv\" (UID: \"f40225b3-db40-4aa8-a465-98a998aa9549\") " pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"
Nov 03 21:12:36.915138 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:36.915069    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c5c7\" (UniqueName: \"kubernetes.io/projected/f40225b3-db40-4aa8-a465-98a998aa9549-kube-api-access-6c5c7\") pod \"e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv\" (UID: \"f40225b3-db40-4aa8-a465-98a998aa9549\") " pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"
Nov 03 21:12:36.915138 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:36.915128    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-model-cache\") pod \"e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv\" (UID: \"f40225b3-db40-4aa8-a465-98a998aa9549\") " pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"
Nov 03 21:12:36.915200 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:36.915172    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f40225b3-db40-4aa8-a465-98a998aa9549-tls-certs\") pod \"e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv\" (UID: \"f40225b3-db40-4aa8-a465-98a998aa9549\") " pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"
Nov 03 21:12:37.016409 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:37.016367    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-model-cache\") pod \"e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv\" (UID: \"f40225b3-db40-4aa8-a465-98a998aa9549\") " pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"
Nov 03 21:12:37.016647 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:37.016429    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f40225b3-db40-4aa8-a465-98a998aa9549-tls-certs\") pod \"e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv\" (UID: \"f40225b3-db40-4aa8-a465-98a998aa9549\") " pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"
Nov 03 21:12:37.016647 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:37.016467    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-tmp-dir\") pod \"e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv\" (UID: \"f40225b3-db40-4aa8-a465-98a998aa9549\") " pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"
Nov 03 21:12:37.016647 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:37.016510    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-dshm\") pod \"e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv\" (UID: \"f40225b3-db40-4aa8-a465-98a998aa9549\") " pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"
Nov 03 21:12:37.016647 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:37.016535    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-home\") pod \"e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv\" (UID: \"f40225b3-db40-4aa8-a465-98a998aa9549\") " pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"
Nov 03 21:12:37.016647 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:37.016630    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c5c7\" (UniqueName: \"kubernetes.io/projected/f40225b3-db40-4aa8-a465-98a998aa9549-kube-api-access-6c5c7\") pod \"e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv\" (UID: \"f40225b3-db40-4aa8-a465-98a998aa9549\") " pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"
Nov 03 21:12:37.016935 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:37.016867    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-model-cache\") pod \"e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv\" (UID: \"f40225b3-db40-4aa8-a465-98a998aa9549\") " pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"
Nov 03 21:12:37.016935 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:37.016905    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-tmp-dir\") pod \"e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv\" (UID: \"f40225b3-db40-4aa8-a465-98a998aa9549\") " pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"
Nov 03 21:12:37.017017 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:37.016940    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-home\") pod \"e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv\" (UID: \"f40225b3-db40-4aa8-a465-98a998aa9549\") " pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"
Nov 03 21:12:37.019008 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:37.018988    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-dshm\") pod \"e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv\" (UID: \"f40225b3-db40-4aa8-a465-98a998aa9549\") " pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"
Nov 03 21:12:37.019122 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:37.019082    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f40225b3-db40-4aa8-a465-98a998aa9549-tls-certs\") pod \"e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv\" (UID: \"f40225b3-db40-4aa8-a465-98a998aa9549\") " pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"
Nov 03 21:12:37.029283 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:37.029260    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c5c7\" (UniqueName: \"kubernetes.io/projected/f40225b3-db40-4aa8-a465-98a998aa9549-kube-api-access-6c5c7\") pod \"e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv\" (UID: \"f40225b3-db40-4aa8-a465-98a998aa9549\") " pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"
Nov 03 21:12:37.137229 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:37.137138    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"
Nov 03 21:12:37.412449 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:37.412368    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"]
Nov 03 21:12:37.416088 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:12:37.416059    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf40225b3_db40_4aa8_a465_98a998aa9549.slice/crio-2c55be470e5455e7b0ed57468b4dc49b06406087ae0daa168fb510b8c14005ad WatchSource:0}: Error finding container 2c55be470e5455e7b0ed57468b4dc49b06406087ae0daa168fb510b8c14005ad: Status 404 returned error can't find the container with id 2c55be470e5455e7b0ed57468b4dc49b06406087ae0daa168fb510b8c14005ad
Nov 03 21:12:38.397117 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:38.397080    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv" event={"ID":"f40225b3-db40-4aa8-a465-98a998aa9549","Type":"ContainerStarted","Data":"c49f2692e604765e2717e212010811d666312e9a7f5bb976d008ad16dd7417e2"}
Nov 03 21:12:38.397117 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:38.397120    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv" event={"ID":"f40225b3-db40-4aa8-a465-98a998aa9549","Type":"ContainerStarted","Data":"2c55be470e5455e7b0ed57468b4dc49b06406087ae0daa168fb510b8c14005ad"}
Nov 03 21:12:38.417682 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:38.417463    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv" podStartSLOduration=2.4174481070000002 podStartE2EDuration="2.417448107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:12:38.417135029 +0000 UTC m=+4288.550263494" watchObservedRunningTime="2026-11-03 21:12:38.417448107 +0000 UTC m=+4288.550576570"
Nov 03 21:12:39.137865 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:39.137830    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"
Nov 03 21:12:42.505622 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:42.504722    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-6bhmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:12:42.505622 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:42.504806    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr" podUID="9d6e2ee7-7d48-4879-b6fa-f303b5d24697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:12:47.137759 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:47.137719    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"
Nov 03 21:12:47.150502 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:47.150478    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"
Nov 03 21:12:47.487976 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:47.487943    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"
Nov 03 21:12:49.627339 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:12:49.627301    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:12:52.430462 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:52.430430    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"]
Nov 03 21:12:52.433904 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:52.433887    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"
Nov 03 21:12:52.449542 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:52.449513    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-29948937-kserve-self-signed-certs\""
Nov 03 21:12:52.462797 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:52.462772    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"]
Nov 03 21:12:52.516827 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:52.516795    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-tls-certs\") pod \"e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc\" (UID: \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\") " pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"
Nov 03 21:12:52.516984 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:52.516831    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-tmp-dir\") pod \"e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc\" (UID: \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\") " pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"
Nov 03 21:12:52.516984 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:52.516909    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4qld\" (UniqueName: \"kubernetes.io/projected/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-kube-api-access-x4qld\") pod \"e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc\" (UID: \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\") " pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"
Nov 03 21:12:52.517066 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:52.517046    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-dshm\") pod \"e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc\" (UID: \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\") " pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"
Nov 03 21:12:52.517108 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:52.517093    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-model-cache\") pod \"e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc\" (UID: \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\") " pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"
Nov 03 21:12:52.517151 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:52.517137    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-home\") pod \"e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc\" (UID: \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\") " pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"
Nov 03 21:12:52.618322 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:52.618288    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4qld\" (UniqueName: \"kubernetes.io/projected/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-kube-api-access-x4qld\") pod \"e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc\" (UID: \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\") " pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"
Nov 03 21:12:52.618502 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:52.618405    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-dshm\") pod \"e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc\" (UID: \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\") " pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"
Nov 03 21:12:52.618502 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:52.618437    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-model-cache\") pod \"e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc\" (UID: \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\") " pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"
Nov 03 21:12:52.618502 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:52.618474    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-home\") pod \"e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc\" (UID: \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\") " pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"
Nov 03 21:12:52.618502 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:52.618500    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-tls-certs\") pod \"e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc\" (UID: \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\") " pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"
Nov 03 21:12:52.618751 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:52.618527    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-tmp-dir\") pod \"e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc\" (UID: \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\") " pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"
Nov 03 21:12:52.618930 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:52.618898    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-home\") pod \"e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc\" (UID: \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\") " pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"
Nov 03 21:12:52.619064 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:52.618932    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-model-cache\") pod \"e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc\" (UID: \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\") " pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"
Nov 03 21:12:52.619064 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:52.618989    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-tmp-dir\") pod \"e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc\" (UID: \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\") " pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"
Nov 03 21:12:52.620876 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:52.620855    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-dshm\") pod \"e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc\" (UID: \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\") " pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"
Nov 03 21:12:52.621042 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:52.621024    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-tls-certs\") pod \"e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc\" (UID: \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\") " pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"
Nov 03 21:12:52.630899 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:52.630877    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4qld\" (UniqueName: \"kubernetes.io/projected/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-kube-api-access-x4qld\") pod \"e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc\" (UID: \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\") " pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"
Nov 03 21:12:52.745090 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:52.745065    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"
Nov 03 21:12:53.020161 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:53.020136    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"]
Nov 03 21:12:53.022620 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:12:53.022577    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42cfb39c_fcd1_488a_ab09_cfb65b07b6bc.slice/crio-43eef9f33769e90691670da69cada4fa594764bfa69acdf9dabcafcd3f861989 WatchSource:0}: Error finding container 43eef9f33769e90691670da69cada4fa594764bfa69acdf9dabcafcd3f861989: Status 404 returned error can't find the container with id 43eef9f33769e90691670da69cada4fa594764bfa69acdf9dabcafcd3f861989
Nov 03 21:12:53.506639 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:53.506569    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc" event={"ID":"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc","Type":"ContainerStarted","Data":"af77324e53b4a169423d2d0b3dc6a259cb966cc57ef3914294bb35bb428d69df"}
Nov 03 21:12:53.507069 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:53.506656    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc" event={"ID":"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc","Type":"ContainerStarted","Data":"43eef9f33769e90691670da69cada4fa594764bfa69acdf9dabcafcd3f861989"}
Nov 03 21:12:53.527369 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:53.527322    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc" podStartSLOduration=1.5273092099999999 podStartE2EDuration="1.52730921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:12:53.525390652 +0000 UTC m=+4303.658519115" watchObservedRunningTime="2026-11-03 21:12:53.52730921 +0000 UTC m=+4303.660437671"
Nov 03 21:12:53.745835 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:53.745799    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"
Nov 03 21:12:57.485747 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:57.485702    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-6bhmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:12:57.486398 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:57.486362    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr" podUID="9d6e2ee7-7d48-4879-b6fa-f303b5d24697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:12:57.924676 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:57.923758    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-cd5ll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:12:57.924676 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:57.923824    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll" podUID="b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:12:58.603623 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:58.602711    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bnt6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:12:58.603623 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:12:58.602778    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k" podUID="9375775d-7b1d-48d0-ab7b-ce8cf03729e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:13:02.745904 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:02.745864    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"
Nov 03 21:13:02.758078 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:02.758044    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"
Nov 03 21:13:03.577225 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:03.577197    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"
Nov 03 21:13:12.931093 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:12.930706    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-cd5ll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:13:12.931093 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:12.930788    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll" podUID="b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:13:13.601949 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:13.601910    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bnt6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:13:13.602374 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:13.602343    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k" podUID="9375775d-7b1d-48d0-ab7b-ce8cf03729e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:13:19.628713 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:13:19.628669    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:13:20.224800 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:20.224765    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"]
Nov 03 21:13:22.464954 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:22.464789    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"]
Nov 03 21:13:28.603746 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:28.603703    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bnt6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:13:28.604403 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:28.604367    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k" podUID="9375775d-7b1d-48d0-ab7b-ce8cf03729e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:13:35.245655 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.245617    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv" podUID="f40225b3-db40-4aa8-a465-98a998aa9549" containerName="main" containerID="cri-o://c49f2692e604765e2717e212010811d666312e9a7f5bb976d008ad16dd7417e2" gracePeriod=45
Nov 03 21:13:35.495089 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.495059    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"
Nov 03 21:13:35.568408 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.568311    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-dshm\") pod \"f40225b3-db40-4aa8-a465-98a998aa9549\" (UID: \"f40225b3-db40-4aa8-a465-98a998aa9549\") "
Nov 03 21:13:35.568630 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.568410    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f40225b3-db40-4aa8-a465-98a998aa9549-tls-certs\" (UniqueName: \"kubernetes.io/secret/f40225b3-db40-4aa8-a465-98a998aa9549-tls-certs\") pod \"f40225b3-db40-4aa8-a465-98a998aa9549\" (UID: \"f40225b3-db40-4aa8-a465-98a998aa9549\") "
Nov 03 21:13:35.568630 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.568439    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-tmp-dir\") pod \"f40225b3-db40-4aa8-a465-98a998aa9549\" (UID: \"f40225b3-db40-4aa8-a465-98a998aa9549\") "
Nov 03 21:13:35.568630 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.568460    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-home\" (UniqueName: \"kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-home\") pod \"f40225b3-db40-4aa8-a465-98a998aa9549\" (UID: \"f40225b3-db40-4aa8-a465-98a998aa9549\") "
Nov 03 21:13:35.568630 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.568486    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-model-cache\") pod \"f40225b3-db40-4aa8-a465-98a998aa9549\" (UID: \"f40225b3-db40-4aa8-a465-98a998aa9549\") "
Nov 03 21:13:35.568630 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.568522    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40225b3-db40-4aa8-a465-98a998aa9549-kube-api-access-6c5c7\" (UniqueName: \"kubernetes.io/projected/f40225b3-db40-4aa8-a465-98a998aa9549-kube-api-access-6c5c7\") pod \"f40225b3-db40-4aa8-a465-98a998aa9549\" (UID: \"f40225b3-db40-4aa8-a465-98a998aa9549\") "
Nov 03 21:13:35.568943 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.568846    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-tmp-dir" pod "f40225b3-db40-4aa8-a465-98a998aa9549" (UID: "f40225b3-db40-4aa8-a465-98a998aa9549"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:13:35.568943 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.568861    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-model-cache" pod "f40225b3-db40-4aa8-a465-98a998aa9549" (UID: "f40225b3-db40-4aa8-a465-98a998aa9549"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:13:35.568943 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.568871    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-home" pod "f40225b3-db40-4aa8-a465-98a998aa9549" (UID: "f40225b3-db40-4aa8-a465-98a998aa9549"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:13:35.569116 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.568963    4421 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-model-cache\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:35.569116 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.568986    4421 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-tmp-dir\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:35.570760 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.570732    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-dshm" pod "f40225b3-db40-4aa8-a465-98a998aa9549" (UID: "f40225b3-db40-4aa8-a465-98a998aa9549"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:13:35.570917 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.570896    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40225b3-db40-4aa8-a465-98a998aa9549-kube-api-access-6c5c7" pod "f40225b3-db40-4aa8-a465-98a998aa9549" (UID: "f40225b3-db40-4aa8-a465-98a998aa9549"). InnerVolumeSpecName "kube-api-access-6c5c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:13:35.571129 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.571113    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f40225b3-db40-4aa8-a465-98a998aa9549-tls-certs" pod "f40225b3-db40-4aa8-a465-98a998aa9549" (UID: "f40225b3-db40-4aa8-a465-98a998aa9549"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:13:35.670361 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.670320    4421 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f40225b3-db40-4aa8-a465-98a998aa9549-tls-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:35.670361 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.670353    4421 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-home\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:35.670361 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.670363    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c5c7\" (UniqueName: \"kubernetes.io/projected/f40225b3-db40-4aa8-a465-98a998aa9549-kube-api-access-6c5c7\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:35.670658 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.670373    4421 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f40225b3-db40-4aa8-a465-98a998aa9549-dshm\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:35.803080 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.803043    4421 generic.go:356] "Generic (PLEG): container finished" podID="f40225b3-db40-4aa8-a465-98a998aa9549" containerID="c49f2692e604765e2717e212010811d666312e9a7f5bb976d008ad16dd7417e2" exitCode=0
Nov 03 21:13:35.803294 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.803101    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"
Nov 03 21:13:35.803294 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.803100    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv" event={"ID":"f40225b3-db40-4aa8-a465-98a998aa9549","Type":"ContainerDied","Data":"c49f2692e604765e2717e212010811d666312e9a7f5bb976d008ad16dd7417e2"}
Nov 03 21:13:35.803294 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.803223    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv" event={"ID":"f40225b3-db40-4aa8-a465-98a998aa9549","Type":"ContainerDied","Data":"2c55be470e5455e7b0ed57468b4dc49b06406087ae0daa168fb510b8c14005ad"}
Nov 03 21:13:35.803294 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.803254    4421 scope.go:122] "RemoveContainer" containerID="c49f2692e604765e2717e212010811d666312e9a7f5bb976d008ad16dd7417e2"
Nov 03 21:13:35.817007 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.816980    4421 scope.go:122] "RemoveContainer" containerID="c49f2692e604765e2717e212010811d666312e9a7f5bb976d008ad16dd7417e2"
Nov 03 21:13:35.817483 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:13:35.817452    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c49f2692e604765e2717e212010811d666312e9a7f5bb976d008ad16dd7417e2\": container with ID starting with c49f2692e604765e2717e212010811d666312e9a7f5bb976d008ad16dd7417e2 not found: ID does not exist" containerID="c49f2692e604765e2717e212010811d666312e9a7f5bb976d008ad16dd7417e2"
Nov 03 21:13:35.817648 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.817492    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49f2692e604765e2717e212010811d666312e9a7f5bb976d008ad16dd7417e2"} err="failed to get container status \"c49f2692e604765e2717e212010811d666312e9a7f5bb976d008ad16dd7417e2\": rpc error: code = NotFound desc = could not find container \"c49f2692e604765e2717e212010811d666312e9a7f5bb976d008ad16dd7417e2\": container with ID starting with c49f2692e604765e2717e212010811d666312e9a7f5bb976d008ad16dd7417e2 not found: ID does not exist"
Nov 03 21:13:35.827186 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.827134    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"]
Nov 03 21:13:35.833272 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:35.833252    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv"]
Nov 03 21:13:36.013920 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:36.013890    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f40225b3-db40-4aa8-a465-98a998aa9549" path="/var/lib/kubelet/pods/f40225b3-db40-4aa8-a465-98a998aa9549/volumes"
Nov 03 21:13:36.478111 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:36.478066    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-29948937-kserve-85d88f7bb9-j5ztv" podUID="f40225b3-db40-4aa8-a465-98a998aa9549" containerName="main" probeResult="failure" output="Get \"https://10.134.1.14:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:13:37.496512 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:37.496475    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc" podUID="42cfb39c-fcd1-488a-ab09-cfb65b07b6bc" containerName="main" containerID="cri-o://af77324e53b4a169423d2d0b3dc6a259cb966cc57ef3914294bb35bb428d69df" gracePeriod=45
Nov 03 21:13:37.567400 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:37.567364    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc" podUID="42cfb39c-fcd1-488a-ab09-cfb65b07b6bc" containerName="main" probeResult="failure" output="Get \"https://10.134.1.15:8000/ready\": dial tcp 10.134.1.15:8000: connect: connection refused"
Nov 03 21:13:37.770661 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:37.770577    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"
Nov 03 21:13:37.817274 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:37.817241    4421 generic.go:356] "Generic (PLEG): container finished" podID="42cfb39c-fcd1-488a-ab09-cfb65b07b6bc" containerID="af77324e53b4a169423d2d0b3dc6a259cb966cc57ef3914294bb35bb428d69df" exitCode=0
Nov 03 21:13:37.817475 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:37.817311    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"
Nov 03 21:13:37.817563 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:37.817331    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc" event={"ID":"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc","Type":"ContainerDied","Data":"af77324e53b4a169423d2d0b3dc6a259cb966cc57ef3914294bb35bb428d69df"}
Nov 03 21:13:37.817746 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:37.817717    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc" event={"ID":"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc","Type":"ContainerDied","Data":"43eef9f33769e90691670da69cada4fa594764bfa69acdf9dabcafcd3f861989"}
Nov 03 21:13:37.817915 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:37.817756    4421 scope.go:122] "RemoveContainer" containerID="af77324e53b4a169423d2d0b3dc6a259cb966cc57ef3914294bb35bb428d69df"
Nov 03 21:13:37.893841 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:37.887780    4421 scope.go:122] "RemoveContainer" containerID="af77324e53b4a169423d2d0b3dc6a259cb966cc57ef3914294bb35bb428d69df"
Nov 03 21:13:37.893841 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:13:37.888234    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af77324e53b4a169423d2d0b3dc6a259cb966cc57ef3914294bb35bb428d69df\": container with ID starting with af77324e53b4a169423d2d0b3dc6a259cb966cc57ef3914294bb35bb428d69df not found: ID does not exist" containerID="af77324e53b4a169423d2d0b3dc6a259cb966cc57ef3914294bb35bb428d69df"
Nov 03 21:13:37.893841 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:37.888278    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af77324e53b4a169423d2d0b3dc6a259cb966cc57ef3914294bb35bb428d69df"} err="failed to get container status \"af77324e53b4a169423d2d0b3dc6a259cb966cc57ef3914294bb35bb428d69df\": rpc error: code = NotFound desc = could not find container \"af77324e53b4a169423d2d0b3dc6a259cb966cc57ef3914294bb35bb428d69df\": container with ID starting with af77324e53b4a169423d2d0b3dc6a259cb966cc57ef3914294bb35bb428d69df not found: ID does not exist"
Nov 03 21:13:37.914624 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:37.910915    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-kube-api-access-x4qld\" (UniqueName: \"kubernetes.io/projected/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-kube-api-access-x4qld\") pod \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\" (UID: \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\") "
Nov 03 21:13:37.914624 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:37.911005    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-tls-certs\" (UniqueName: \"kubernetes.io/secret/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-tls-certs\") pod \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\" (UID: \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\") "
Nov 03 21:13:37.914624 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:37.911076    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-home\" (UniqueName: \"kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-home\") pod \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\" (UID: \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\") "
Nov 03 21:13:37.914624 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:37.911144    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-tmp-dir\") pod \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\" (UID: \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\") "
Nov 03 21:13:37.914624 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:37.911190    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-model-cache\") pod \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\" (UID: \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\") "
Nov 03 21:13:37.914624 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:37.911257    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-dshm\") pod \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\" (UID: \"42cfb39c-fcd1-488a-ab09-cfb65b07b6bc\") "
Nov 03 21:13:37.914624 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:37.912194    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-home" pod "42cfb39c-fcd1-488a-ab09-cfb65b07b6bc" (UID: "42cfb39c-fcd1-488a-ab09-cfb65b07b6bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:13:37.914624 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:37.913996    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-tmp-dir" pod "42cfb39c-fcd1-488a-ab09-cfb65b07b6bc" (UID: "42cfb39c-fcd1-488a-ab09-cfb65b07b6bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:13:37.914624 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:37.914182    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-model-cache" pod "42cfb39c-fcd1-488a-ab09-cfb65b07b6bc" (UID: "42cfb39c-fcd1-488a-ab09-cfb65b07b6bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:13:37.929297 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:37.929269    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-dshm" pod "42cfb39c-fcd1-488a-ab09-cfb65b07b6bc" (UID: "42cfb39c-fcd1-488a-ab09-cfb65b07b6bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:13:37.929720 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:37.929687    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-kube-api-access-x4qld" pod "42cfb39c-fcd1-488a-ab09-cfb65b07b6bc" (UID: "42cfb39c-fcd1-488a-ab09-cfb65b07b6bc"). InnerVolumeSpecName "kube-api-access-x4qld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:13:37.931439 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:37.931394    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-tls-certs" pod "42cfb39c-fcd1-488a-ab09-cfb65b07b6bc" (UID: "42cfb39c-fcd1-488a-ab09-cfb65b07b6bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:13:38.017637 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:38.014259    4421 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-model-cache\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:38.017637 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:38.014293    4421 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-dshm\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:38.017637 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:38.014313    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4qld\" (UniqueName: \"kubernetes.io/projected/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-kube-api-access-x4qld\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:38.017637 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:38.014329    4421 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-tls-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:38.017637 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:38.014342    4421 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-home\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:38.017637 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:38.014381    4421 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc-tmp-dir\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:13:38.153635 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:38.148895    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"]
Nov 03 21:13:38.158062 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:38.158037    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-29948937-kserve-5bff757f8f-gs2sc"]
Nov 03 21:13:40.018624 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:40.018225    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42cfb39c-fcd1-488a-ab09-cfb65b07b6bc" path="/var/lib/kubelet/pods/42cfb39c-fcd1-488a-ab09-cfb65b07b6bc/volumes"
Nov 03 21:13:42.490735 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:42.490698    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-6bhmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:13:42.491381 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:42.491349    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr" podUID="9d6e2ee7-7d48-4879-b6fa-f303b5d24697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:13:42.940621 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:42.937691    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-cd5ll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:13:42.940621 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:42.937761    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll" podUID="b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:13:49.627881 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:13:49.627800    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:13:57.766194 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:57.766156    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-99f95578b-gg9x5"]
Nov 03 21:13:57.766667 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:57.766551    4421 cpu_manager.go:414] "containerMap: removing container" podUID="42cfb39c-fcd1-488a-ab09-cfb65b07b6bc" containerName="main"
Nov 03 21:13:57.766667 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:57.766562    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42cfb39c-fcd1-488a-ab09-cfb65b07b6bc" containerName="main"
Nov 03 21:13:57.766667 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:57.766581    4421 cpu_manager.go:414] "containerMap: removing container" podUID="f40225b3-db40-4aa8-a465-98a998aa9549" containerName="main"
Nov 03 21:13:57.766667 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:57.766585    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f40225b3-db40-4aa8-a465-98a998aa9549" containerName="main"
Nov 03 21:13:57.766667 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:57.766659    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="42cfb39c-fcd1-488a-ab09-cfb65b07b6bc" containerName="main"
Nov 03 21:13:57.766667 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:57.766670    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="f40225b3-db40-4aa8-a465-98a998aa9549" containerName="main"
Nov 03 21:13:57.769388 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:57.769371    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-99f95578b-gg9x5"
Nov 03 21:13:57.779626 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:57.779582    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-99f95578b-gg9x5"]
Nov 03 21:13:57.909356 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:57.909313    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kqcz\" (UniqueName: \"kubernetes.io/projected/0b922626-5715-484e-9a34-53d9c24ff25a-kube-api-access-2kqcz\") pod \"payload-pre-processing-99f95578b-gg9x5\" (UID: \"0b922626-5715-484e-9a34-53d9c24ff25a\") " pod="openshift-ingress/payload-pre-processing-99f95578b-gg9x5"
Nov 03 21:13:57.909540 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:57.909373    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b922626-5715-484e-9a34-53d9c24ff25a-plugins-config-volume\") pod \"payload-pre-processing-99f95578b-gg9x5\" (UID: \"0b922626-5715-484e-9a34-53d9c24ff25a\") " pod="openshift-ingress/payload-pre-processing-99f95578b-gg9x5"
Nov 03 21:13:58.010014 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:58.009977    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kqcz\" (UniqueName: \"kubernetes.io/projected/0b922626-5715-484e-9a34-53d9c24ff25a-kube-api-access-2kqcz\") pod \"payload-pre-processing-99f95578b-gg9x5\" (UID: \"0b922626-5715-484e-9a34-53d9c24ff25a\") " pod="openshift-ingress/payload-pre-processing-99f95578b-gg9x5"
Nov 03 21:13:58.010218 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:58.010036    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b922626-5715-484e-9a34-53d9c24ff25a-plugins-config-volume\") pod \"payload-pre-processing-99f95578b-gg9x5\" (UID: \"0b922626-5715-484e-9a34-53d9c24ff25a\") " pod="openshift-ingress/payload-pre-processing-99f95578b-gg9x5"
Nov 03 21:13:58.010985 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:58.010964    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b922626-5715-484e-9a34-53d9c24ff25a-plugins-config-volume\") pod \"payload-pre-processing-99f95578b-gg9x5\" (UID: \"0b922626-5715-484e-9a34-53d9c24ff25a\") " pod="openshift-ingress/payload-pre-processing-99f95578b-gg9x5"
Nov 03 21:13:58.023115 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:58.023060    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kqcz\" (UniqueName: \"kubernetes.io/projected/0b922626-5715-484e-9a34-53d9c24ff25a-kube-api-access-2kqcz\") pod \"payload-pre-processing-99f95578b-gg9x5\" (UID: \"0b922626-5715-484e-9a34-53d9c24ff25a\") " pod="openshift-ingress/payload-pre-processing-99f95578b-gg9x5"
Nov 03 21:13:58.084249 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:58.084215    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-99f95578b-gg9x5"
Nov 03 21:13:58.357218 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:58.357196    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-99f95578b-gg9x5"]
Nov 03 21:13:58.360039 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:13:58.360013    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b922626_5715_484e_9a34_53d9c24ff25a.slice/crio-cde09f0f63af10dfbec14024c9d9d7c5852e108543985caf751f5630253a3494 WatchSource:0}: Error finding container cde09f0f63af10dfbec14024c9d9d7c5852e108543985caf751f5630253a3494: Status 404 returned error can't find the container with id cde09f0f63af10dfbec14024c9d9d7c5852e108543985caf751f5630253a3494
Nov 03 21:13:58.973309 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:58.973271    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-99f95578b-gg9x5" event={"ID":"0b922626-5715-484e-9a34-53d9c24ff25a","Type":"ContainerStarted","Data":"55f60bd1cab704c4a18ecbed51f5436e1288caaaa909b793035c3a3855f0afe2"}
Nov 03 21:13:58.973751 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:58.973327    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-99f95578b-gg9x5" event={"ID":"0b922626-5715-484e-9a34-53d9c24ff25a","Type":"ContainerStarted","Data":"cde09f0f63af10dfbec14024c9d9d7c5852e108543985caf751f5630253a3494"}
Nov 03 21:13:58.973751 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:58.973360    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-99f95578b-gg9x5"
Nov 03 21:13:58.992489 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:13:58.992442    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-99f95578b-gg9x5" podStartSLOduration=1.992427336 podStartE2EDuration="1.992427336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:13:58.992242158 +0000 UTC m=+4369.125370620" watchObservedRunningTime="2026-11-03 21:13:58.992427336 +0000 UTC m=+4369.125555798"
Nov 03 21:14:09.980382 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:09.980353    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-99f95578b-gg9x5"
Nov 03 21:14:11.767148 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:11.767103    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-57dd5dc5bc-7z6kd"]
Nov 03 21:14:11.770777 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:11.770760    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-57dd5dc5bc-7z6kd"
Nov 03 21:14:11.782046 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:11.782019    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-57dd5dc5bc-7z6kd"]
Nov 03 21:14:11.831609 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:11.831562    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/726f44a9-f258-4142-a16f-782042829754-plugins-config-volume\") pod \"payload-processing-57dd5dc5bc-7z6kd\" (UID: \"726f44a9-f258-4142-a16f-782042829754\") " pod="openshift-ingress/payload-processing-57dd5dc5bc-7z6kd"
Nov 03 21:14:11.831787 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:11.831631    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhb2m\" (UniqueName: \"kubernetes.io/projected/726f44a9-f258-4142-a16f-782042829754-kube-api-access-jhb2m\") pod \"payload-processing-57dd5dc5bc-7z6kd\" (UID: \"726f44a9-f258-4142-a16f-782042829754\") " pod="openshift-ingress/payload-processing-57dd5dc5bc-7z6kd"
Nov 03 21:14:11.932751 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:11.932716    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/726f44a9-f258-4142-a16f-782042829754-plugins-config-volume\") pod \"payload-processing-57dd5dc5bc-7z6kd\" (UID: \"726f44a9-f258-4142-a16f-782042829754\") " pod="openshift-ingress/payload-processing-57dd5dc5bc-7z6kd"
Nov 03 21:14:11.932947 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:11.932777    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhb2m\" (UniqueName: \"kubernetes.io/projected/726f44a9-f258-4142-a16f-782042829754-kube-api-access-jhb2m\") pod \"payload-processing-57dd5dc5bc-7z6kd\" (UID: \"726f44a9-f258-4142-a16f-782042829754\") " pod="openshift-ingress/payload-processing-57dd5dc5bc-7z6kd"
Nov 03 21:14:11.933413 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:11.933391    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/726f44a9-f258-4142-a16f-782042829754-plugins-config-volume\") pod \"payload-processing-57dd5dc5bc-7z6kd\" (UID: \"726f44a9-f258-4142-a16f-782042829754\") " pod="openshift-ingress/payload-processing-57dd5dc5bc-7z6kd"
Nov 03 21:14:11.945686 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:11.945658    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhb2m\" (UniqueName: \"kubernetes.io/projected/726f44a9-f258-4142-a16f-782042829754-kube-api-access-jhb2m\") pod \"payload-processing-57dd5dc5bc-7z6kd\" (UID: \"726f44a9-f258-4142-a16f-782042829754\") " pod="openshift-ingress/payload-processing-57dd5dc5bc-7z6kd"
Nov 03 21:14:12.084350 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:12.084247    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-57dd5dc5bc-7z6kd"
Nov 03 21:14:12.361198 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:12.361139    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-57dd5dc5bc-7z6kd"]
Nov 03 21:14:12.363301 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:14:12.363274    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod726f44a9_f258_4142_a16f_782042829754.slice/crio-a4e4d71a6b2831b555447c35d14ed72d32b7afc79e63c281720943841166bd0b WatchSource:0}: Error finding container a4e4d71a6b2831b555447c35d14ed72d32b7afc79e63c281720943841166bd0b: Status 404 returned error can't find the container with id a4e4d71a6b2831b555447c35d14ed72d32b7afc79e63c281720943841166bd0b
Nov 03 21:14:13.040493 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:13.040458    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-57dd5dc5bc-7z6kd" event={"ID":"726f44a9-f258-4142-a16f-782042829754","Type":"ContainerStarted","Data":"23f4170cdc12871a78277c93a7553db8313e5e7c99186f492de4b91a0e9c2181"}
Nov 03 21:14:13.040938 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:13.040511    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-57dd5dc5bc-7z6kd" event={"ID":"726f44a9-f258-4142-a16f-782042829754","Type":"ContainerStarted","Data":"a4e4d71a6b2831b555447c35d14ed72d32b7afc79e63c281720943841166bd0b"}
Nov 03 21:14:13.040938 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:13.040575    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-57dd5dc5bc-7z6kd"
Nov 03 21:14:13.059846 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:13.059796    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-57dd5dc5bc-7z6kd" podStartSLOduration=2.059783057 podStartE2EDuration="2.059783057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:13.058257868 +0000 UTC m=+4383.191386331" watchObservedRunningTime="2026-11-03 21:14:13.059783057 +0000 UTC m=+4383.192911519"
Nov 03 21:14:19.627577 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:14:19.627537    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:14:24.048433 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:24.048399    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-57dd5dc5bc-7z6kd"
Nov 03 21:14:24.097439 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:24.097393    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-xx9xb"]
Nov 03 21:14:26.005333 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:26.005292    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7dc89df4c4-hhqqf"]
Nov 03 21:14:26.009099 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:26.009073    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7dc89df4c4-hhqqf"
Nov 03 21:14:26.018046 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:26.018023    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7dc89df4c4-hhqqf"]
Nov 03 21:14:26.064241 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:26.064201    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhrks\" (UniqueName: \"kubernetes.io/projected/e78b2b21-5706-465b-8ea1-9f1679e0f4ff-kube-api-access-zhrks\") pod \"payload-processing-7dc89df4c4-hhqqf\" (UID: \"e78b2b21-5706-465b-8ea1-9f1679e0f4ff\") " pod="openshift-ingress/payload-processing-7dc89df4c4-hhqqf"
Nov 03 21:14:26.064411 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:26.064318    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e78b2b21-5706-465b-8ea1-9f1679e0f4ff-plugins-config-volume\") pod \"payload-processing-7dc89df4c4-hhqqf\" (UID: \"e78b2b21-5706-465b-8ea1-9f1679e0f4ff\") " pod="openshift-ingress/payload-processing-7dc89df4c4-hhqqf"
Nov 03 21:14:26.165500 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:26.165464    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhrks\" (UniqueName: \"kubernetes.io/projected/e78b2b21-5706-465b-8ea1-9f1679e0f4ff-kube-api-access-zhrks\") pod \"payload-processing-7dc89df4c4-hhqqf\" (UID: \"e78b2b21-5706-465b-8ea1-9f1679e0f4ff\") " pod="openshift-ingress/payload-processing-7dc89df4c4-hhqqf"
Nov 03 21:14:26.165735 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:26.165552    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e78b2b21-5706-465b-8ea1-9f1679e0f4ff-plugins-config-volume\") pod \"payload-processing-7dc89df4c4-hhqqf\" (UID: \"e78b2b21-5706-465b-8ea1-9f1679e0f4ff\") " pod="openshift-ingress/payload-processing-7dc89df4c4-hhqqf"
Nov 03 21:14:26.166367 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:26.166343    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e78b2b21-5706-465b-8ea1-9f1679e0f4ff-plugins-config-volume\") pod \"payload-processing-7dc89df4c4-hhqqf\" (UID: \"e78b2b21-5706-465b-8ea1-9f1679e0f4ff\") " pod="openshift-ingress/payload-processing-7dc89df4c4-hhqqf"
Nov 03 21:14:26.176729 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:26.176704    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-5556d898cf-4gn6l"]
Nov 03 21:14:26.180292 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:26.180274    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5556d898cf-4gn6l"
Nov 03 21:14:26.181691 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:26.181668    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhrks\" (UniqueName: \"kubernetes.io/projected/e78b2b21-5706-465b-8ea1-9f1679e0f4ff-kube-api-access-zhrks\") pod \"payload-processing-7dc89df4c4-hhqqf\" (UID: \"e78b2b21-5706-465b-8ea1-9f1679e0f4ff\") " pod="openshift-ingress/payload-processing-7dc89df4c4-hhqqf"
Nov 03 21:14:26.188958 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:26.188935    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-5556d898cf-4gn6l"]
Nov 03 21:14:26.266724 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:26.266642    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835ed6d1-8b5c-4b49-8316-0fbb1d161e2c-plugins-config-volume\") pod \"payload-pre-processing-5556d898cf-4gn6l\" (UID: \"835ed6d1-8b5c-4b49-8316-0fbb1d161e2c\") " pod="openshift-ingress/payload-pre-processing-5556d898cf-4gn6l"
Nov 03 21:14:26.266874 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:26.266767    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bnsl\" (UniqueName: \"kubernetes.io/projected/835ed6d1-8b5c-4b49-8316-0fbb1d161e2c-kube-api-access-7bnsl\") pod \"payload-pre-processing-5556d898cf-4gn6l\" (UID: \"835ed6d1-8b5c-4b49-8316-0fbb1d161e2c\") " pod="openshift-ingress/payload-pre-processing-5556d898cf-4gn6l"
Nov 03 21:14:26.324609 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:26.324565    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7dc89df4c4-hhqqf"
Nov 03 21:14:26.368106 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:26.368074    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835ed6d1-8b5c-4b49-8316-0fbb1d161e2c-plugins-config-volume\") pod \"payload-pre-processing-5556d898cf-4gn6l\" (UID: \"835ed6d1-8b5c-4b49-8316-0fbb1d161e2c\") " pod="openshift-ingress/payload-pre-processing-5556d898cf-4gn6l"
Nov 03 21:14:26.368272 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:26.368182    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bnsl\" (UniqueName: \"kubernetes.io/projected/835ed6d1-8b5c-4b49-8316-0fbb1d161e2c-kube-api-access-7bnsl\") pod \"payload-pre-processing-5556d898cf-4gn6l\" (UID: \"835ed6d1-8b5c-4b49-8316-0fbb1d161e2c\") " pod="openshift-ingress/payload-pre-processing-5556d898cf-4gn6l"
Nov 03 21:14:26.368791 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:26.368772    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835ed6d1-8b5c-4b49-8316-0fbb1d161e2c-plugins-config-volume\") pod \"payload-pre-processing-5556d898cf-4gn6l\" (UID: \"835ed6d1-8b5c-4b49-8316-0fbb1d161e2c\") " pod="openshift-ingress/payload-pre-processing-5556d898cf-4gn6l"
Nov 03 21:14:26.382633 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:26.382591    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bnsl\" (UniqueName: \"kubernetes.io/projected/835ed6d1-8b5c-4b49-8316-0fbb1d161e2c-kube-api-access-7bnsl\") pod \"payload-pre-processing-5556d898cf-4gn6l\" (UID: \"835ed6d1-8b5c-4b49-8316-0fbb1d161e2c\") " pod="openshift-ingress/payload-pre-processing-5556d898cf-4gn6l"
Nov 03 21:14:26.504906 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:26.504871    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5556d898cf-4gn6l"
Nov 03 21:14:26.603335 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:26.603310    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7dc89df4c4-hhqqf"]
Nov 03 21:14:26.605749 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:14:26.605719    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78b2b21_5706_465b_8ea1_9f1679e0f4ff.slice/crio-7040dbcbab44197ad1fc4e5ceff1dd57b331dbc8a6e83ed0211671ef5068cd90 WatchSource:0}: Error finding container 7040dbcbab44197ad1fc4e5ceff1dd57b331dbc8a6e83ed0211671ef5068cd90: Status 404 returned error can't find the container with id 7040dbcbab44197ad1fc4e5ceff1dd57b331dbc8a6e83ed0211671ef5068cd90
Nov 03 21:14:26.778500 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:26.778424    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-5556d898cf-4gn6l"]
Nov 03 21:14:26.781303 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:14:26.781272    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835ed6d1_8b5c_4b49_8316_0fbb1d161e2c.slice/crio-ea6371be34586b1b54284b1292ea11c5c0fc05ec11caa2389daf4f912ab66cd9 WatchSource:0}: Error finding container ea6371be34586b1b54284b1292ea11c5c0fc05ec11caa2389daf4f912ab66cd9: Status 404 returned error can't find the container with id ea6371be34586b1b54284b1292ea11c5c0fc05ec11caa2389daf4f912ab66cd9
Nov 03 21:14:27.115927 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:27.115832    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7dc89df4c4-hhqqf" event={"ID":"e78b2b21-5706-465b-8ea1-9f1679e0f4ff","Type":"ContainerStarted","Data":"d7e092d7e7b9f8770df12b211479bf3ba47e92f4b0fec65276b99e8145b6452d"}
Nov 03 21:14:27.115927 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:27.115890    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7dc89df4c4-hhqqf" event={"ID":"e78b2b21-5706-465b-8ea1-9f1679e0f4ff","Type":"ContainerStarted","Data":"7040dbcbab44197ad1fc4e5ceff1dd57b331dbc8a6e83ed0211671ef5068cd90"}
Nov 03 21:14:27.115927 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:27.115915    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7dc89df4c4-hhqqf"
Nov 03 21:14:27.117452 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:27.117429    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5556d898cf-4gn6l" event={"ID":"835ed6d1-8b5c-4b49-8316-0fbb1d161e2c","Type":"ContainerStarted","Data":"6971c61876e5bad87704536c9ba012f64d5fd19d8559bb57325d2f26b5df7c31"}
Nov 03 21:14:27.117635 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:27.117466    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5556d898cf-4gn6l" event={"ID":"835ed6d1-8b5c-4b49-8316-0fbb1d161e2c","Type":"ContainerStarted","Data":"ea6371be34586b1b54284b1292ea11c5c0fc05ec11caa2389daf4f912ab66cd9"}
Nov 03 21:14:27.117635 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:27.117550    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-5556d898cf-4gn6l"
Nov 03 21:14:27.136641 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:27.136574    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7dc89df4c4-hhqqf" podStartSLOduration=2.136561169 podStartE2EDuration="2.136561169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:27.133733087 +0000 UTC m=+4397.266861559" watchObservedRunningTime="2026-11-03 21:14:27.136561169 +0000 UTC m=+4397.269689630"
Nov 03 21:14:27.152364 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:27.152316    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-5556d898cf-4gn6l" podStartSLOduration=1.152302669 podStartE2EDuration="1.152302669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:27.151453498 +0000 UTC m=+4397.284581963" watchObservedRunningTime="2026-11-03 21:14:27.152302669 +0000 UTC m=+4397.285431130"
Nov 03 21:14:29.098324 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:29.098263    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-7bb9489676-xx9xb" podUID="79a63610-3e85-4ddd-a7a8-6ceccc2b3f83" containerName="payload-processing" containerID="cri-o://a782baadb7c5adbcf90db6c6c017abe4da2c8fb4d6762b2af08a3d5bb4be49d6" gracePeriod=25
Nov 03 21:14:29.106321 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:14:29.106283    4421 logging.go:55] [core] [Channel #19193 SubChannel #19194]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.31:50052", ServerName: "10.134.0.31:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.31:50052: connect: connection refused"
Nov 03 21:14:29.357667 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:29.357585    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-xx9xb"
Nov 03 21:14:29.397846 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:29.397797    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79a63610-3e85-4ddd-a7a8-6ceccc2b3f83-kube-api-access-smxs8\" (UniqueName: \"kubernetes.io/projected/79a63610-3e85-4ddd-a7a8-6ceccc2b3f83-kube-api-access-smxs8\") pod \"79a63610-3e85-4ddd-a7a8-6ceccc2b3f83\" (UID: \"79a63610-3e85-4ddd-a7a8-6ceccc2b3f83\") "
Nov 03 21:14:29.398015 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:29.397939    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79a63610-3e85-4ddd-a7a8-6ceccc2b3f83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79a63610-3e85-4ddd-a7a8-6ceccc2b3f83-plugins-config-volume\") pod \"79a63610-3e85-4ddd-a7a8-6ceccc2b3f83\" (UID: \"79a63610-3e85-4ddd-a7a8-6ceccc2b3f83\") "
Nov 03 21:14:29.398331 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:29.398299    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79a63610-3e85-4ddd-a7a8-6ceccc2b3f83-plugins-config-volume" pod "79a63610-3e85-4ddd-a7a8-6ceccc2b3f83" (UID: "79a63610-3e85-4ddd-a7a8-6ceccc2b3f83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:14:29.400067 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:29.400046    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79a63610-3e85-4ddd-a7a8-6ceccc2b3f83-kube-api-access-smxs8" pod "79a63610-3e85-4ddd-a7a8-6ceccc2b3f83" (UID: "79a63610-3e85-4ddd-a7a8-6ceccc2b3f83"). InnerVolumeSpecName "kube-api-access-smxs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:14:29.499008 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:29.498968    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smxs8\" (UniqueName: \"kubernetes.io/projected/79a63610-3e85-4ddd-a7a8-6ceccc2b3f83-kube-api-access-smxs8\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:29.499008 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:29.499001    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79a63610-3e85-4ddd-a7a8-6ceccc2b3f83-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:30.106543 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:30.106510    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-7bb9489676-xx9xb" podUID="79a63610-3e85-4ddd-a7a8-6ceccc2b3f83" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.31:50052\" within 1s: context deadline exceeded"
Nov 03 21:14:30.134308 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:30.134278    4421 generic.go:356] "Generic (PLEG): container finished" podID="79a63610-3e85-4ddd-a7a8-6ceccc2b3f83" containerID="a782baadb7c5adbcf90db6c6c017abe4da2c8fb4d6762b2af08a3d5bb4be49d6" exitCode=0
Nov 03 21:14:30.134465 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:30.134327    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-xx9xb"
Nov 03 21:14:30.134465 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:30.134362    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-xx9xb" event={"ID":"79a63610-3e85-4ddd-a7a8-6ceccc2b3f83","Type":"ContainerDied","Data":"a782baadb7c5adbcf90db6c6c017abe4da2c8fb4d6762b2af08a3d5bb4be49d6"}
Nov 03 21:14:30.134465 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:30.134419    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-xx9xb" event={"ID":"79a63610-3e85-4ddd-a7a8-6ceccc2b3f83","Type":"ContainerDied","Data":"e71c87a9998d5c082e7ede6d6cee018ff46039852d062b3c9437f19f8fa48f99"}
Nov 03 21:14:30.134465 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:30.134447    4421 scope.go:122] "RemoveContainer" containerID="a782baadb7c5adbcf90db6c6c017abe4da2c8fb4d6762b2af08a3d5bb4be49d6"
Nov 03 21:14:30.143547 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:30.143530    4421 scope.go:122] "RemoveContainer" containerID="a782baadb7c5adbcf90db6c6c017abe4da2c8fb4d6762b2af08a3d5bb4be49d6"
Nov 03 21:14:30.143844 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:14:30.143824    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a782baadb7c5adbcf90db6c6c017abe4da2c8fb4d6762b2af08a3d5bb4be49d6\": container with ID starting with a782baadb7c5adbcf90db6c6c017abe4da2c8fb4d6762b2af08a3d5bb4be49d6 not found: ID does not exist" containerID="a782baadb7c5adbcf90db6c6c017abe4da2c8fb4d6762b2af08a3d5bb4be49d6"
Nov 03 21:14:30.143894 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:30.143851    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a782baadb7c5adbcf90db6c6c017abe4da2c8fb4d6762b2af08a3d5bb4be49d6"} err="failed to get container status \"a782baadb7c5adbcf90db6c6c017abe4da2c8fb4d6762b2af08a3d5bb4be49d6\": rpc error: code = NotFound desc = could not find container \"a782baadb7c5adbcf90db6c6c017abe4da2c8fb4d6762b2af08a3d5bb4be49d6\": container with ID starting with a782baadb7c5adbcf90db6c6c017abe4da2c8fb4d6762b2af08a3d5bb4be49d6 not found: ID does not exist"
Nov 03 21:14:30.151901 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:30.151878    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-xx9xb"]
Nov 03 21:14:30.156037 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:30.156018    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-xx9xb"]
Nov 03 21:14:32.014987 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:32.014957    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79a63610-3e85-4ddd-a7a8-6ceccc2b3f83" path="/var/lib/kubelet/pods/79a63610-3e85-4ddd-a7a8-6ceccc2b3f83/volumes"
Nov 03 21:14:38.128313 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:38.128287    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-5556d898cf-4gn6l"
Nov 03 21:14:38.128708 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:38.128349    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7dc89df4c4-hhqqf"
Nov 03 21:14:38.178886 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:38.178849    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-99f95578b-gg9x5"]
Nov 03 21:14:38.198076 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:38.198045    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-57dd5dc5bc-7z6kd"]
Nov 03 21:14:40.866981 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:40.866943    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4"]
Nov 03 21:14:40.867442 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:40.867393    4421 cpu_manager.go:414] "containerMap: removing container" podUID="79a63610-3e85-4ddd-a7a8-6ceccc2b3f83" containerName="payload-processing"
Nov 03 21:14:40.867555 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:40.867446    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79a63610-3e85-4ddd-a7a8-6ceccc2b3f83" containerName="payload-processing"
Nov 03 21:14:40.867681 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:40.867656    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="79a63610-3e85-4ddd-a7a8-6ceccc2b3f83" containerName="payload-processing"
Nov 03 21:14:40.906048 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:40.906001    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4"]
Nov 03 21:14:40.906255 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:40.906143    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4"
Nov 03 21:14:40.910282 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:40.910246    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\""
Nov 03 21:14:40.910282 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:40.910278    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 21:14:40.915669 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:40.915383    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-pcrxj\""
Nov 03 21:14:41.010556 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:41.010526    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 21:14:41.010810 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:41.010526    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 21:14:41.011414 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:41.011379    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 21:14:41.013177 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:41.013149    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qwhb\" (UniqueName: \"kubernetes.io/projected/133a6445-e5fd-4aad-a9a9-04bbeab5e403-kube-api-access-4qwhb\") pod \"payload-processing-external-model-7fd6967f4c-gwsc4\" (UID: \"133a6445-e5fd-4aad-a9a9-04bbeab5e403\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4"
Nov 03 21:14:41.013290 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:41.013198    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/133a6445-e5fd-4aad-a9a9-04bbeab5e403-provider-credentials\") pod \"payload-processing-external-model-7fd6967f4c-gwsc4\" (UID: \"133a6445-e5fd-4aad-a9a9-04bbeab5e403\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4"
Nov 03 21:14:41.013290 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:41.013270    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/133a6445-e5fd-4aad-a9a9-04bbeab5e403-routing-overlay-volume\") pod \"payload-processing-external-model-7fd6967f4c-gwsc4\" (UID: \"133a6445-e5fd-4aad-a9a9-04bbeab5e403\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4"
Nov 03 21:14:41.013422 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:41.013399    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/133a6445-e5fd-4aad-a9a9-04bbeab5e403-plugins-config-volume\") pod \"payload-processing-external-model-7fd6967f4c-gwsc4\" (UID: \"133a6445-e5fd-4aad-a9a9-04bbeab5e403\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4"
Nov 03 21:14:41.114280 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:41.114240    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qwhb\" (UniqueName: \"kubernetes.io/projected/133a6445-e5fd-4aad-a9a9-04bbeab5e403-kube-api-access-4qwhb\") pod \"payload-processing-external-model-7fd6967f4c-gwsc4\" (UID: \"133a6445-e5fd-4aad-a9a9-04bbeab5e403\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4"
Nov 03 21:14:41.114497 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:41.114305    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/133a6445-e5fd-4aad-a9a9-04bbeab5e403-provider-credentials\") pod \"payload-processing-external-model-7fd6967f4c-gwsc4\" (UID: \"133a6445-e5fd-4aad-a9a9-04bbeab5e403\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4"
Nov 03 21:14:41.114497 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:41.114331    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/133a6445-e5fd-4aad-a9a9-04bbeab5e403-routing-overlay-volume\") pod \"payload-processing-external-model-7fd6967f4c-gwsc4\" (UID: \"133a6445-e5fd-4aad-a9a9-04bbeab5e403\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4"
Nov 03 21:14:41.114497 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:41.114465    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/133a6445-e5fd-4aad-a9a9-04bbeab5e403-plugins-config-volume\") pod \"payload-processing-external-model-7fd6967f4c-gwsc4\" (UID: \"133a6445-e5fd-4aad-a9a9-04bbeab5e403\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4"
Nov 03 21:14:41.115308 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:41.115282    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/133a6445-e5fd-4aad-a9a9-04bbeab5e403-routing-overlay-volume\") pod \"payload-processing-external-model-7fd6967f4c-gwsc4\" (UID: \"133a6445-e5fd-4aad-a9a9-04bbeab5e403\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4"
Nov 03 21:14:41.115535 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:41.115510    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/133a6445-e5fd-4aad-a9a9-04bbeab5e403-plugins-config-volume\") pod \"payload-processing-external-model-7fd6967f4c-gwsc4\" (UID: \"133a6445-e5fd-4aad-a9a9-04bbeab5e403\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4"
Nov 03 21:14:41.117276 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:41.117222    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/133a6445-e5fd-4aad-a9a9-04bbeab5e403-provider-credentials\") pod \"payload-processing-external-model-7fd6967f4c-gwsc4\" (UID: \"133a6445-e5fd-4aad-a9a9-04bbeab5e403\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4"
Nov 03 21:14:41.128281 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:41.128261    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qwhb\" (UniqueName: \"kubernetes.io/projected/133a6445-e5fd-4aad-a9a9-04bbeab5e403-kube-api-access-4qwhb\") pod \"payload-processing-external-model-7fd6967f4c-gwsc4\" (UID: \"133a6445-e5fd-4aad-a9a9-04bbeab5e403\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4"
Nov 03 21:14:41.221994 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:41.221964    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4"
Nov 03 21:14:41.502325 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:41.502299    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4"]
Nov 03 21:14:41.504276 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:14:41.504247    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod133a6445_e5fd_4aad_a9a9_04bbeab5e403.slice/crio-f3985b4eb3700c0108ad1eba3b0dc456ef3b887c26de76242d8bf6c0c99d4de3 WatchSource:0}: Error finding container f3985b4eb3700c0108ad1eba3b0dc456ef3b887c26de76242d8bf6c0c99d4de3: Status 404 returned error can't find the container with id f3985b4eb3700c0108ad1eba3b0dc456ef3b887c26de76242d8bf6c0c99d4de3
Nov 03 21:14:42.188814 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:42.188781    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4" event={"ID":"133a6445-e5fd-4aad-a9a9-04bbeab5e403","Type":"ContainerStarted","Data":"82f36056124246c29ad07aa4e2dbe292419978425f5e3b89067f40168dc9d11b"}
Nov 03 21:14:42.189238 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:42.188830    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4" event={"ID":"133a6445-e5fd-4aad-a9a9-04bbeab5e403","Type":"ContainerStarted","Data":"f3985b4eb3700c0108ad1eba3b0dc456ef3b887c26de76242d8bf6c0c99d4de3"}
Nov 03 21:14:42.189238 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:42.188880    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4"
Nov 03 21:14:42.207788 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:42.207734    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4" podStartSLOduration=2.207718857 podStartE2EDuration="2.207718857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:42.206953897 +0000 UTC m=+4412.340082362" watchObservedRunningTime="2026-11-03 21:14:42.207718857 +0000 UTC m=+4412.340847314"
Nov 03 21:14:43.180126 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:43.180063    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-99f95578b-gg9x5" podUID="0b922626-5715-484e-9a34-53d9c24ff25a" containerName="payload-pre-processing" containerID="cri-o://55f60bd1cab704c4a18ecbed51f5436e1288caaaa909b793035c3a3855f0afe2" gracePeriod=25
Nov 03 21:14:43.199298 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:43.199236    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-57dd5dc5bc-7z6kd" podUID="726f44a9-f258-4142-a16f-782042829754" containerName="payload-processing" containerID="cri-o://23f4170cdc12871a78277c93a7553db8313e5e7c99186f492de4b91a0e9c2181" gracePeriod=25
Nov 03 21:14:43.542427 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:43.542385    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-99f95578b-gg9x5"
Nov 03 21:14:43.569492 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:43.567678    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-57dd5dc5bc-7z6kd"
Nov 03 21:14:43.646198 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:43.645773    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/726f44a9-f258-4142-a16f-782042829754-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/726f44a9-f258-4142-a16f-782042829754-plugins-config-volume\") pod \"726f44a9-f258-4142-a16f-782042829754\" (UID: \"726f44a9-f258-4142-a16f-782042829754\") "
Nov 03 21:14:43.646198 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:43.645889    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b922626-5715-484e-9a34-53d9c24ff25a-kube-api-access-2kqcz\" (UniqueName: \"kubernetes.io/projected/0b922626-5715-484e-9a34-53d9c24ff25a-kube-api-access-2kqcz\") pod \"0b922626-5715-484e-9a34-53d9c24ff25a\" (UID: \"0b922626-5715-484e-9a34-53d9c24ff25a\") "
Nov 03 21:14:43.646198 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:43.645928    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/726f44a9-f258-4142-a16f-782042829754-kube-api-access-jhb2m\" (UniqueName: \"kubernetes.io/projected/726f44a9-f258-4142-a16f-782042829754-kube-api-access-jhb2m\") pod \"726f44a9-f258-4142-a16f-782042829754\" (UID: \"726f44a9-f258-4142-a16f-782042829754\") "
Nov 03 21:14:43.646198 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:43.646047    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b922626-5715-484e-9a34-53d9c24ff25a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b922626-5715-484e-9a34-53d9c24ff25a-plugins-config-volume\") pod \"0b922626-5715-484e-9a34-53d9c24ff25a\" (UID: \"0b922626-5715-484e-9a34-53d9c24ff25a\") "
Nov 03 21:14:43.646613 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:43.646255    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/726f44a9-f258-4142-a16f-782042829754-plugins-config-volume" pod "726f44a9-f258-4142-a16f-782042829754" (UID: "726f44a9-f258-4142-a16f-782042829754"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:14:43.646613 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:43.646404    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/726f44a9-f258-4142-a16f-782042829754-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:43.646753 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:43.646703    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b922626-5715-484e-9a34-53d9c24ff25a-plugins-config-volume" pod "0b922626-5715-484e-9a34-53d9c24ff25a" (UID: "0b922626-5715-484e-9a34-53d9c24ff25a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:14:43.649410 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:43.649386    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/726f44a9-f258-4142-a16f-782042829754-kube-api-access-jhb2m" pod "726f44a9-f258-4142-a16f-782042829754" (UID: "726f44a9-f258-4142-a16f-782042829754"). InnerVolumeSpecName "kube-api-access-jhb2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:14:43.649632 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:43.649497    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b922626-5715-484e-9a34-53d9c24ff25a-kube-api-access-2kqcz" pod "0b922626-5715-484e-9a34-53d9c24ff25a" (UID: "0b922626-5715-484e-9a34-53d9c24ff25a"). InnerVolumeSpecName "kube-api-access-2kqcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:14:43.748210 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:43.748108    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kqcz\" (UniqueName: \"kubernetes.io/projected/0b922626-5715-484e-9a34-53d9c24ff25a-kube-api-access-2kqcz\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:43.748210 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:43.748150    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhb2m\" (UniqueName: \"kubernetes.io/projected/726f44a9-f258-4142-a16f-782042829754-kube-api-access-jhb2m\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:43.748210 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:43.748165    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b922626-5715-484e-9a34-53d9c24ff25a-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:14:44.230619 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:44.227653    4421 generic.go:356] "Generic (PLEG): container finished" podID="726f44a9-f258-4142-a16f-782042829754" containerID="23f4170cdc12871a78277c93a7553db8313e5e7c99186f492de4b91a0e9c2181" exitCode=0
Nov 03 21:14:44.230619 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:44.227844    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-57dd5dc5bc-7z6kd" event={"ID":"726f44a9-f258-4142-a16f-782042829754","Type":"ContainerDied","Data":"23f4170cdc12871a78277c93a7553db8313e5e7c99186f492de4b91a0e9c2181"}
Nov 03 21:14:44.230619 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:44.227899    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-57dd5dc5bc-7z6kd" event={"ID":"726f44a9-f258-4142-a16f-782042829754","Type":"ContainerDied","Data":"a4e4d71a6b2831b555447c35d14ed72d32b7afc79e63c281720943841166bd0b"}
Nov 03 21:14:44.230619 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:44.227930    4421 scope.go:122] "RemoveContainer" containerID="23f4170cdc12871a78277c93a7553db8313e5e7c99186f492de4b91a0e9c2181"
Nov 03 21:14:44.230619 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:44.228098    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-57dd5dc5bc-7z6kd"
Nov 03 21:14:44.235007 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:44.232874    4421 generic.go:356] "Generic (PLEG): container finished" podID="0b922626-5715-484e-9a34-53d9c24ff25a" containerID="55f60bd1cab704c4a18ecbed51f5436e1288caaaa909b793035c3a3855f0afe2" exitCode=0
Nov 03 21:14:44.235007 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:44.232929    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-99f95578b-gg9x5" event={"ID":"0b922626-5715-484e-9a34-53d9c24ff25a","Type":"ContainerDied","Data":"55f60bd1cab704c4a18ecbed51f5436e1288caaaa909b793035c3a3855f0afe2"}
Nov 03 21:14:44.235007 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:44.232972    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-99f95578b-gg9x5" event={"ID":"0b922626-5715-484e-9a34-53d9c24ff25a","Type":"ContainerDied","Data":"cde09f0f63af10dfbec14024c9d9d7c5852e108543985caf751f5630253a3494"}
Nov 03 21:14:44.235007 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:44.233048    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-99f95578b-gg9x5"
Nov 03 21:14:44.292896 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:44.292131    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-99f95578b-gg9x5"]
Nov 03 21:14:44.292896 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:44.292278    4421 scope.go:122] "RemoveContainer" containerID="23f4170cdc12871a78277c93a7553db8313e5e7c99186f492de4b91a0e9c2181"
Nov 03 21:14:44.322955 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:14:44.320749    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23f4170cdc12871a78277c93a7553db8313e5e7c99186f492de4b91a0e9c2181\": container with ID starting with 23f4170cdc12871a78277c93a7553db8313e5e7c99186f492de4b91a0e9c2181 not found: ID does not exist" containerID="23f4170cdc12871a78277c93a7553db8313e5e7c99186f492de4b91a0e9c2181"
Nov 03 21:14:44.322955 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:44.320798    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23f4170cdc12871a78277c93a7553db8313e5e7c99186f492de4b91a0e9c2181"} err="failed to get container status \"23f4170cdc12871a78277c93a7553db8313e5e7c99186f492de4b91a0e9c2181\": rpc error: code = NotFound desc = could not find container \"23f4170cdc12871a78277c93a7553db8313e5e7c99186f492de4b91a0e9c2181\": container with ID starting with 23f4170cdc12871a78277c93a7553db8313e5e7c99186f492de4b91a0e9c2181 not found: ID does not exist"
Nov 03 21:14:44.322955 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:44.320826    4421 scope.go:122] "RemoveContainer" containerID="55f60bd1cab704c4a18ecbed51f5436e1288caaaa909b793035c3a3855f0afe2"
Nov 03 21:14:44.337162 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:44.336891    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-99f95578b-gg9x5"]
Nov 03 21:14:44.421633 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:44.420652    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-57dd5dc5bc-7z6kd"]
Nov 03 21:14:44.439623 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:44.424925    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-57dd5dc5bc-7z6kd"]
Nov 03 21:14:44.495299 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:44.494779    4421 scope.go:122] "RemoveContainer" containerID="55f60bd1cab704c4a18ecbed51f5436e1288caaaa909b793035c3a3855f0afe2"
Nov 03 21:14:44.495614 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:14:44.495496    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55f60bd1cab704c4a18ecbed51f5436e1288caaaa909b793035c3a3855f0afe2\": container with ID starting with 55f60bd1cab704c4a18ecbed51f5436e1288caaaa909b793035c3a3855f0afe2 not found: ID does not exist" containerID="55f60bd1cab704c4a18ecbed51f5436e1288caaaa909b793035c3a3855f0afe2"
Nov 03 21:14:44.495614 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:44.495535    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55f60bd1cab704c4a18ecbed51f5436e1288caaaa909b793035c3a3855f0afe2"} err="failed to get container status \"55f60bd1cab704c4a18ecbed51f5436e1288caaaa909b793035c3a3855f0afe2\": rpc error: code = NotFound desc = could not find container \"55f60bd1cab704c4a18ecbed51f5436e1288caaaa909b793035c3a3855f0afe2\": container with ID starting with 55f60bd1cab704c4a18ecbed51f5436e1288caaaa909b793035c3a3855f0afe2 not found: ID does not exist"
Nov 03 21:14:46.030694 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:46.030435    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b922626-5715-484e-9a34-53d9c24ff25a" path="/var/lib/kubelet/pods/0b922626-5715-484e-9a34-53d9c24ff25a/volumes"
Nov 03 21:14:46.031215 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:46.031168    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="726f44a9-f258-4142-a16f-782042829754" path="/var/lib/kubelet/pods/726f44a9-f258-4142-a16f-782042829754/volumes"
Nov 03 21:14:46.031534 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:46.031490    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xfwgr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:14:46.031683 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:46.031652    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr" podUID="bc25e3fb-91ba-4d75-92d3-d1aab2feb033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:14:49.627425 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:14:49.627380    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:14:53.199556 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:14:53.199524    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4"
Nov 03 21:15:19.628224 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:15:19.628141    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:15:49.627695 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:15:49.627656    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:16:17.672832 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:17.672795    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4"]
Nov 03 21:16:19.643569 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:16:19.643522    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:16:22.676765 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:22.676686    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4" podUID="133a6445-e5fd-4aad-a9a9-04bbeab5e403" containerName="payload-processing" containerID="cri-o://82f36056124246c29ad07aa4e2dbe292419978425f5e3b89067f40168dc9d11b" gracePeriod=25
Nov 03 21:16:23.208357 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:16:23.208297    4421 logging.go:55] [core] [Channel #19297 SubChannel #19298]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.1.20:50052", ServerName: "10.134.1.20:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.1.20:50052: connect: connection refused"
Nov 03 21:16:23.643803 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:23.643774    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4"
Nov 03 21:16:23.704839 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:23.704808    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/133a6445-e5fd-4aad-a9a9-04bbeab5e403-kube-api-access-4qwhb\" (UniqueName: \"kubernetes.io/projected/133a6445-e5fd-4aad-a9a9-04bbeab5e403-kube-api-access-4qwhb\") pod \"133a6445-e5fd-4aad-a9a9-04bbeab5e403\" (UID: \"133a6445-e5fd-4aad-a9a9-04bbeab5e403\") "
Nov 03 21:16:23.719518 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:23.719493    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/133a6445-e5fd-4aad-a9a9-04bbeab5e403-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/133a6445-e5fd-4aad-a9a9-04bbeab5e403-plugins-config-volume\") pod \"133a6445-e5fd-4aad-a9a9-04bbeab5e403\" (UID: \"133a6445-e5fd-4aad-a9a9-04bbeab5e403\") "
Nov 03 21:16:23.723761 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:23.723734    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/133a6445-e5fd-4aad-a9a9-04bbeab5e403-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/133a6445-e5fd-4aad-a9a9-04bbeab5e403-routing-overlay-volume\") pod \"133a6445-e5fd-4aad-a9a9-04bbeab5e403\" (UID: \"133a6445-e5fd-4aad-a9a9-04bbeab5e403\") "
Nov 03 21:16:23.723965 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:23.723946    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/133a6445-e5fd-4aad-a9a9-04bbeab5e403-provider-credentials\" (UniqueName: \"kubernetes.io/projected/133a6445-e5fd-4aad-a9a9-04bbeab5e403-provider-credentials\") pod \"133a6445-e5fd-4aad-a9a9-04bbeab5e403\" (UID: \"133a6445-e5fd-4aad-a9a9-04bbeab5e403\") "
Nov 03 21:16:23.724743 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:23.723188    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133a6445-e5fd-4aad-a9a9-04bbeab5e403-kube-api-access-4qwhb" pod "133a6445-e5fd-4aad-a9a9-04bbeab5e403" (UID: "133a6445-e5fd-4aad-a9a9-04bbeab5e403"). InnerVolumeSpecName "kube-api-access-4qwhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:23.724878 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:23.723548    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/133a6445-e5fd-4aad-a9a9-04bbeab5e403-plugins-config-volume" pod "133a6445-e5fd-4aad-a9a9-04bbeab5e403" (UID: "133a6445-e5fd-4aad-a9a9-04bbeab5e403"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:16:23.725383 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:23.725362    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/133a6445-e5fd-4aad-a9a9-04bbeab5e403-routing-overlay-volume" pod "133a6445-e5fd-4aad-a9a9-04bbeab5e403" (UID: "133a6445-e5fd-4aad-a9a9-04bbeab5e403"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:16:23.766786 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:23.766725    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133a6445-e5fd-4aad-a9a9-04bbeab5e403-provider-credentials" pod "133a6445-e5fd-4aad-a9a9-04bbeab5e403" (UID: "133a6445-e5fd-4aad-a9a9-04bbeab5e403"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:16:23.796827 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:23.796789    4421 generic.go:356] "Generic (PLEG): container finished" podID="133a6445-e5fd-4aad-a9a9-04bbeab5e403" containerID="82f36056124246c29ad07aa4e2dbe292419978425f5e3b89067f40168dc9d11b" exitCode=0
Nov 03 21:16:23.797127 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:23.797103    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4" event={"ID":"133a6445-e5fd-4aad-a9a9-04bbeab5e403","Type":"ContainerDied","Data":"82f36056124246c29ad07aa4e2dbe292419978425f5e3b89067f40168dc9d11b"}
Nov 03 21:16:23.797290 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:23.797275    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4" event={"ID":"133a6445-e5fd-4aad-a9a9-04bbeab5e403","Type":"ContainerDied","Data":"f3985b4eb3700c0108ad1eba3b0dc456ef3b887c26de76242d8bf6c0c99d4de3"}
Nov 03 21:16:23.797428 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:23.797413    4421 scope.go:122] "RemoveContainer" containerID="82f36056124246c29ad07aa4e2dbe292419978425f5e3b89067f40168dc9d11b"
Nov 03 21:16:23.801750 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:23.801727    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4"
Nov 03 21:16:23.829298 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:23.829271    4421 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/133a6445-e5fd-4aad-a9a9-04bbeab5e403-routing-overlay-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:23.832368 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:23.832351    4421 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/133a6445-e5fd-4aad-a9a9-04bbeab5e403-provider-credentials\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:23.832525 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:23.832509    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qwhb\" (UniqueName: \"kubernetes.io/projected/133a6445-e5fd-4aad-a9a9-04bbeab5e403-kube-api-access-4qwhb\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:23.832687 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:23.832671    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/133a6445-e5fd-4aad-a9a9-04bbeab5e403-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:16:23.872486 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:23.872463    4421 scope.go:122] "RemoveContainer" containerID="82f36056124246c29ad07aa4e2dbe292419978425f5e3b89067f40168dc9d11b"
Nov 03 21:16:23.873060 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:16:23.873037    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82f36056124246c29ad07aa4e2dbe292419978425f5e3b89067f40168dc9d11b\": container with ID starting with 82f36056124246c29ad07aa4e2dbe292419978425f5e3b89067f40168dc9d11b not found: ID does not exist" containerID="82f36056124246c29ad07aa4e2dbe292419978425f5e3b89067f40168dc9d11b"
Nov 03 21:16:23.873215 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:23.873191    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f36056124246c29ad07aa4e2dbe292419978425f5e3b89067f40168dc9d11b"} err="failed to get container status \"82f36056124246c29ad07aa4e2dbe292419978425f5e3b89067f40168dc9d11b\": rpc error: code = NotFound desc = could not find container \"82f36056124246c29ad07aa4e2dbe292419978425f5e3b89067f40168dc9d11b\": container with ID starting with 82f36056124246c29ad07aa4e2dbe292419978425f5e3b89067f40168dc9d11b not found: ID does not exist"
Nov 03 21:16:23.934621 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:23.930797    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4"]
Nov 03 21:16:23.934621 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:23.933887    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4"]
Nov 03 21:16:24.044687 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:24.033902    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="133a6445-e5fd-4aad-a9a9-04bbeab5e403" path="/var/lib/kubelet/pods/133a6445-e5fd-4aad-a9a9-04bbeab5e403/volumes"
Nov 03 21:16:24.207957 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:24.207911    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-gwsc4" podUID="133a6445-e5fd-4aad-a9a9-04bbeab5e403" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.1.20:50052\" within 1s: context deadline exceeded"
Nov 03 21:16:27.503624 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:27.502729    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-6bhmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:16:27.503624 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:27.502802    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr" podUID="9d6e2ee7-7d48-4879-b6fa-f303b5d24697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:16:27.933682 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:27.932758    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-cd5ll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:16:27.933682 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:27.932826    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll" podUID="b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:16:49.627051 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:16:49.626971    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:16:58.607623 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:58.606737    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bnt6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:16:58.607623 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:16:58.606856    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k" podUID="9375775d-7b1d-48d0-ab7b-ce8cf03729e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:17:19.627780 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:17:19.627741    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:17:46.036660 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:17:46.036118    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xfwgr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:17:46.036660 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:17:46.036196    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr" podUID="bc25e3fb-91ba-4d75-92d3-d1aab2feb033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:17:49.627195 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:17:49.627158    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:18:19.628534 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:18:19.628449    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:18:49.627623 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:18:49.627566    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:19:19.627632 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:19:19.627555    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:19:31.031865 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:19:31.031810    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xfwgr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:19:31.032575 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:19:31.032531    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr" podUID="bc25e3fb-91ba-4d75-92d3-d1aab2feb033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:19:49.704684 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:19:49.701842    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:20:16.748142 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.748099    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"]
Nov 03 21:20:16.748895 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.748677    4421 cpu_manager.go:414] "containerMap: removing container" podUID="133a6445-e5fd-4aad-a9a9-04bbeab5e403" containerName="payload-processing"
Nov 03 21:20:16.748895 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.748696    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="133a6445-e5fd-4aad-a9a9-04bbeab5e403" containerName="payload-processing"
Nov 03 21:20:16.748895 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.748722    4421 cpu_manager.go:414] "containerMap: removing container" podUID="0b922626-5715-484e-9a34-53d9c24ff25a" containerName="payload-pre-processing"
Nov 03 21:20:16.748895 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.748728    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b922626-5715-484e-9a34-53d9c24ff25a" containerName="payload-pre-processing"
Nov 03 21:20:16.748895 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.748738    4421 cpu_manager.go:414] "containerMap: removing container" podUID="726f44a9-f258-4142-a16f-782042829754" containerName="payload-processing"
Nov 03 21:20:16.748895 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.748744    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="726f44a9-f258-4142-a16f-782042829754" containerName="payload-processing"
Nov 03 21:20:16.748895 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.748819    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="133a6445-e5fd-4aad-a9a9-04bbeab5e403" containerName="payload-processing"
Nov 03 21:20:16.748895 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.748830    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b922626-5715-484e-9a34-53d9c24ff25a" containerName="payload-pre-processing"
Nov 03 21:20:16.748895 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.748837    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="726f44a9-f258-4142-a16f-782042829754" containerName="payload-processing"
Nov 03 21:20:16.752065 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.752034    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:16.759609 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.759567    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-d9b592-openshift-default-dockercfg-pc5h5\""
Nov 03 21:20:16.768996 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.768968    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"]
Nov 03 21:20:16.836288 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.836248    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6xbr\" (UniqueName: \"kubernetes.io/projected/d668277e-62c6-4f0e-ace6-a6b623f039f2-kube-api-access-x6xbr\") pod \"e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") " pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:16.836531 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.836295    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-workload-socket\") pod \"e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") " pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:16.836531 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.836394    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-token\") pod \"e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") " pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:16.836531 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.836437    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d668277e-62c6-4f0e-ace6-a6b623f039f2-istiod-ca-cert\") pod \"e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") " pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:16.836531 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.836520    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-data\") pod \"e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") " pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:16.836751 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.836565    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-credential-socket\") pod \"e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") " pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:16.836751 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.836710    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-envoy\") pod \"e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") " pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:16.836751 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.836742    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-podinfo\") pod \"e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") " pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:16.836872 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.836769    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-workload-certs\") pod \"e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") " pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:16.937469 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.937423    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d668277e-62c6-4f0e-ace6-a6b623f039f2-istiod-ca-cert\") pod \"e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") " pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:16.937469 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.937473    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-data\") pod \"e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") " pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:16.937794 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.937525    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-credential-socket\") pod \"e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") " pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:16.937794 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.937621    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-envoy\") pod \"e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") " pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:16.937794 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.937662    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-podinfo\") pod \"e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") " pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:16.937794 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.937699    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-workload-certs\") pod \"e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") " pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:16.937794 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.937732    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6xbr\" (UniqueName: \"kubernetes.io/projected/d668277e-62c6-4f0e-ace6-a6b623f039f2-kube-api-access-x6xbr\") pod \"e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") " pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:16.937794 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.937762    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-workload-socket\") pod \"e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") " pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:16.937794 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.937794    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-token\") pod \"e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") " pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:16.938529 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.938497    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-data\") pod \"e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") " pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:16.938694 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.938650    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-workload-certs\") pod \"e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") " pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:16.938772 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.938714    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-workload-socket\") pod \"e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") " pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:16.942271 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.939250    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-credential-socket\") pod \"e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") " pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:16.942271 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.939354    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d668277e-62c6-4f0e-ace6-a6b623f039f2-istiod-ca-cert\") pod \"e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") " pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:16.942271 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.941654    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-envoy\") pod \"e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") " pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:16.949963 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.949939    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-podinfo\") pod \"e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") " pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:16.954091 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.954067    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-token\") pod \"e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") " pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:16.954188 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:16.954104    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6xbr\" (UniqueName: \"kubernetes.io/projected/d668277e-62c6-4f0e-ace6-a6b623f039f2-kube-api-access-x6xbr\") pod \"e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") " pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:17.064902 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:17.064810    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:17.349032 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:17.349004    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"]
Nov 03 21:20:17.350962 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:20:17.350932    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd668277e_62c6_4f0e_ace6_a6b623f039f2.slice/crio-e037b2c82bada0f0af6e1cdcb71cbcadd6a8b47b821b59532d86ab11f3571ece WatchSource:0}: Error finding container e037b2c82bada0f0af6e1cdcb71cbcadd6a8b47b821b59532d86ab11f3571ece: Status 404 returned error can't find the container with id e037b2c82bada0f0af6e1cdcb71cbcadd6a8b47b821b59532d86ab11f3571ece
Nov 03 21:20:17.353080 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:17.353043    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:20:17.353195 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:17.353121    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:20:17.353195 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:17.353166    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:20:18.161494 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:18.161458    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv" event={"ID":"d668277e-62c6-4f0e-ace6-a6b623f039f2","Type":"ContainerStarted","Data":"1d191b6e395a6c4fff1fd4355c23c21721cbe1d6809cb0cfbfd495264f5bc2b9"}
Nov 03 21:20:18.161937 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:18.161519    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv" event={"ID":"d668277e-62c6-4f0e-ace6-a6b623f039f2","Type":"ContainerStarted","Data":"e037b2c82bada0f0af6e1cdcb71cbcadd6a8b47b821b59532d86ab11f3571ece"}
Nov 03 21:20:18.186270 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:18.186214    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv" podStartSLOduration=2.1861961069999998 podStartE2EDuration="2.186196107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:20:18.184677771 +0000 UTC m=+4748.317806233" watchObservedRunningTime="2026-11-03 21:20:18.186196107 +0000 UTC m=+4748.319324570"
Nov 03 21:20:19.065997 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:19.065959    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:19.071075 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:19.071050    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:19.166753 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:19.166718    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:19.167719 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:19.167700    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:19.628622 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:20:19.628560    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:20:21.436252 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.436217    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9"]
Nov 03 21:20:21.440015 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.439990    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9"
Nov 03 21:20:21.445630 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.445587    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-d9b592-serving-cert\""
Nov 03 21:20:21.456018 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.455988    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9"]
Nov 03 21:20:21.485865 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.485827    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgvpd\" (UniqueName: \"kubernetes.io/projected/d0ce49d8-03d5-4881-bb0f-324349505550-kube-api-access-rgvpd\") pod \"maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9\" (UID: \"d0ce49d8-03d5-4881-bb0f-324349505550\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9"
Nov 03 21:20:21.486054 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.485906    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0ce49d8-03d5-4881-bb0f-324349505550-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9\" (UID: \"d0ce49d8-03d5-4881-bb0f-324349505550\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9"
Nov 03 21:20:21.486054 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.485940    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0ce49d8-03d5-4881-bb0f-324349505550-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9\" (UID: \"d0ce49d8-03d5-4881-bb0f-324349505550\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9"
Nov 03 21:20:21.486184 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.486068    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0ce49d8-03d5-4881-bb0f-324349505550-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9\" (UID: \"d0ce49d8-03d5-4881-bb0f-324349505550\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9"
Nov 03 21:20:21.514799 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.514767    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2"]
Nov 03 21:20:21.518209 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.518186    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2"
Nov 03 21:20:21.529085 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.529058    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-d9b592\""
Nov 03 21:20:21.530440 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.530413    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2"]
Nov 03 21:20:21.586532 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.586501    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgvpd\" (UniqueName: \"kubernetes.io/projected/d0ce49d8-03d5-4881-bb0f-324349505550-kube-api-access-rgvpd\") pod \"maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9\" (UID: \"d0ce49d8-03d5-4881-bb0f-324349505550\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9"
Nov 03 21:20:21.586765 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.586554    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0ce49d8-03d5-4881-bb0f-324349505550-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9\" (UID: \"d0ce49d8-03d5-4881-bb0f-324349505550\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9"
Nov 03 21:20:21.586765 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.586733    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0ce49d8-03d5-4881-bb0f-324349505550-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9\" (UID: \"d0ce49d8-03d5-4881-bb0f-324349505550\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9"
Nov 03 21:20:21.586911 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:20:21.586859    4421 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-serving-cert: secret "maas-api-e2e-worker-single-main-d9b592-serving-cert" not found
Nov 03 21:20:21.586911 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.586854    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hwdb\" (UniqueName: \"kubernetes.io/projected/a43cea43-6f73-40c7-b16c-a3502238ab28-kube-api-access-4hwdb\") pod \"payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2\" (UID: \"a43cea43-6f73-40c7-b16c-a3502238ab28\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2"
Nov 03 21:20:21.587019 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.586929    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a43cea43-6f73-40c7-b16c-a3502238ab28-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2\" (UID: \"a43cea43-6f73-40c7-b16c-a3502238ab28\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2"
Nov 03 21:20:21.587019 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:20:21.586975    4421 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0ce49d8-03d5-4881-bb0f-324349505550-maas-api-tls podName:d0ce49d8-03d5-4881-bb0f-324349505550 nodeName:}" failed. No retries permitted until 2026-11-03 21:20:22.086956122 +0000 UTC m=+4752.220084565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d0ce49d8-03d5-4881-bb0f-324349505550-maas-api-tls") pod "maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9" (UID: "d0ce49d8-03d5-4881-bb0f-324349505550") : secret "maas-api-e2e-worker-single-main-d9b592-serving-cert" not found
Nov 03 21:20:21.587019 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.587002    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0ce49d8-03d5-4881-bb0f-324349505550-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9\" (UID: \"d0ce49d8-03d5-4881-bb0f-324349505550\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9"
Nov 03 21:20:21.587540 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.587524    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0ce49d8-03d5-4881-bb0f-324349505550-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9\" (UID: \"d0ce49d8-03d5-4881-bb0f-324349505550\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9"
Nov 03 21:20:21.589500 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.589476    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0ce49d8-03d5-4881-bb0f-324349505550-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9\" (UID: \"d0ce49d8-03d5-4881-bb0f-324349505550\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9"
Nov 03 21:20:21.592191 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.592166    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl"]
Nov 03 21:20:21.595806 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.595789    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl"
Nov 03 21:20:21.606050 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.606027    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgvpd\" (UniqueName: \"kubernetes.io/projected/d0ce49d8-03d5-4881-bb0f-324349505550-kube-api-access-rgvpd\") pod \"maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9\" (UID: \"d0ce49d8-03d5-4881-bb0f-324349505550\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9"
Nov 03 21:20:21.606645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.606624    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl"]
Nov 03 21:20:21.607838 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.607421    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-4ad4ba24-dockercfg-k2p75\""
Nov 03 21:20:21.689576 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.689464    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcb2f012-348e-4395-97f3-eed7277d44fd-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl\" (UID: \"bcb2f012-348e-4395-97f3-eed7277d44fd\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl"
Nov 03 21:20:21.689576 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.689520    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7bqz\" (UniqueName: \"kubernetes.io/projected/bcb2f012-348e-4395-97f3-eed7277d44fd-kube-api-access-s7bqz\") pod \"payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl\" (UID: \"bcb2f012-348e-4395-97f3-eed7277d44fd\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl"
Nov 03 21:20:21.689895 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.689851    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hwdb\" (UniqueName: \"kubernetes.io/projected/a43cea43-6f73-40c7-b16c-a3502238ab28-kube-api-access-4hwdb\") pod \"payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2\" (UID: \"a43cea43-6f73-40c7-b16c-a3502238ab28\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2"
Nov 03 21:20:21.689982 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.689917    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a43cea43-6f73-40c7-b16c-a3502238ab28-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2\" (UID: \"a43cea43-6f73-40c7-b16c-a3502238ab28\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2"
Nov 03 21:20:21.691099 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.691047    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a43cea43-6f73-40c7-b16c-a3502238ab28-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2\" (UID: \"a43cea43-6f73-40c7-b16c-a3502238ab28\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2"
Nov 03 21:20:21.702484 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.702457    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hwdb\" (UniqueName: \"kubernetes.io/projected/a43cea43-6f73-40c7-b16c-a3502238ab28-kube-api-access-4hwdb\") pod \"payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2\" (UID: \"a43cea43-6f73-40c7-b16c-a3502238ab28\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2"
Nov 03 21:20:21.790837 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.790797    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcb2f012-348e-4395-97f3-eed7277d44fd-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl\" (UID: \"bcb2f012-348e-4395-97f3-eed7277d44fd\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl"
Nov 03 21:20:21.790837 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.790837    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7bqz\" (UniqueName: \"kubernetes.io/projected/bcb2f012-348e-4395-97f3-eed7277d44fd-kube-api-access-s7bqz\") pod \"payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl\" (UID: \"bcb2f012-348e-4395-97f3-eed7277d44fd\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl"
Nov 03 21:20:21.791535 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.791508    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcb2f012-348e-4395-97f3-eed7277d44fd-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl\" (UID: \"bcb2f012-348e-4395-97f3-eed7277d44fd\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl"
Nov 03 21:20:21.802903 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.802882    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7bqz\" (UniqueName: \"kubernetes.io/projected/bcb2f012-348e-4395-97f3-eed7277d44fd-kube-api-access-s7bqz\") pod \"payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl\" (UID: \"bcb2f012-348e-4395-97f3-eed7277d44fd\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl"
Nov 03 21:20:21.833757 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.833733    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2"
Nov 03 21:20:21.923413 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:21.923388    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl"
Nov 03 21:20:22.094366 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:22.094327    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0ce49d8-03d5-4881-bb0f-324349505550-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9\" (UID: \"d0ce49d8-03d5-4881-bb0f-324349505550\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9"
Nov 03 21:20:22.096975 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:22.096951    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0ce49d8-03d5-4881-bb0f-324349505550-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9\" (UID: \"d0ce49d8-03d5-4881-bb0f-324349505550\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9"
Nov 03 21:20:22.111802 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:22.111783    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2"]
Nov 03 21:20:22.114534 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:20:22.114509    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda43cea43_6f73_40c7_b16c_a3502238ab28.slice/crio-85f6c7220db19710608b86e9b07e1c653e528b5b2bf3ea499e38802786d03c35 WatchSource:0}: Error finding container 85f6c7220db19710608b86e9b07e1c653e528b5b2bf3ea499e38802786d03c35: Status 404 returned error can't find the container with id 85f6c7220db19710608b86e9b07e1c653e528b5b2bf3ea499e38802786d03c35
Nov 03 21:20:22.181517 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:22.181485    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2" event={"ID":"a43cea43-6f73-40c7-b16c-a3502238ab28","Type":"ContainerStarted","Data":"85f6c7220db19710608b86e9b07e1c653e528b5b2bf3ea499e38802786d03c35"}
Nov 03 21:20:22.202955 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:22.202924    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl"]
Nov 03 21:20:22.205045 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:20:22.205018    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb2f012_348e_4395_97f3_eed7277d44fd.slice/crio-89869cd53fd16cf7b6e30e6c1a4cca710fcd58994b5c022f53ee1c31207b108a WatchSource:0}: Error finding container 89869cd53fd16cf7b6e30e6c1a4cca710fcd58994b5c022f53ee1c31207b108a: Status 404 returned error can't find the container with id 89869cd53fd16cf7b6e30e6c1a4cca710fcd58994b5c022f53ee1c31207b108a
Nov 03 21:20:22.357830 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:22.357792    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9"
Nov 03 21:20:22.637694 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:22.637622    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9"]
Nov 03 21:20:22.641454 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:20:22.641429    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ce49d8_03d5_4881_bb0f_324349505550.slice/crio-52b9e881a501d2c4b52a4a43d84bb57a2038dcf316a9478fe49f6d5620041870 WatchSource:0}: Error finding container 52b9e881a501d2c4b52a4a43d84bb57a2038dcf316a9478fe49f6d5620041870: Status 404 returned error can't find the container with id 52b9e881a501d2c4b52a4a43d84bb57a2038dcf316a9478fe49f6d5620041870
Nov 03 21:20:22.642926 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:22.642908    4421 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:20:23.187526 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:23.187492    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl" event={"ID":"bcb2f012-348e-4395-97f3-eed7277d44fd","Type":"ContainerStarted","Data":"3c4b239946fd41f5772f7847a0c47b02dd0b88bc3e405cb4bb2259c0b37ff9a2"}
Nov 03 21:20:23.187731 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:23.187541    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl" event={"ID":"bcb2f012-348e-4395-97f3-eed7277d44fd","Type":"ContainerStarted","Data":"89869cd53fd16cf7b6e30e6c1a4cca710fcd58994b5c022f53ee1c31207b108a"}
Nov 03 21:20:23.187731 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:23.187643    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl"
Nov 03 21:20:23.188792 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:23.188770    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9" event={"ID":"d0ce49d8-03d5-4881-bb0f-324349505550","Type":"ContainerStarted","Data":"52b9e881a501d2c4b52a4a43d84bb57a2038dcf316a9478fe49f6d5620041870"}
Nov 03 21:20:23.190211 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:23.190191    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2" event={"ID":"a43cea43-6f73-40c7-b16c-a3502238ab28","Type":"ContainerStarted","Data":"cd40e1e48e2c8f59ba6efea2d238015b36dc5dacb057e2b996b81fee90e7d5fd"}
Nov 03 21:20:23.190798 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:23.190316    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2"
Nov 03 21:20:23.207411 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:23.207364    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl" podStartSLOduration=2.207352805 podStartE2EDuration="2.207352805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:20:23.20507411 +0000 UTC m=+4753.338202574" watchObservedRunningTime="2026-11-03 21:20:23.207352805 +0000 UTC m=+4753.340481264"
Nov 03 21:20:23.222527 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:23.222486    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2" podStartSLOduration=2.222475782 podStartE2EDuration="2.222475782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:20:23.22235684 +0000 UTC m=+4753.355485302" watchObservedRunningTime="2026-11-03 21:20:23.222475782 +0000 UTC m=+4753.355604243"
Nov 03 21:20:24.201136 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:24.201097    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9" event={"ID":"d0ce49d8-03d5-4881-bb0f-324349505550","Type":"ContainerStarted","Data":"2d6ddd3c966fd7e2364c2eef9609e3f8940b70f4159f233d22969f3e9ca07e7a"}
Nov 03 21:20:24.201632 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:24.201585    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9"
Nov 03 21:20:24.222943 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:24.222886    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9" podStartSLOduration=2.675234325 podStartE2EDuration="3.222869136s" totalImagesPullingTime="547.634811ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:20:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:20:24.22236996 +0000 UTC m=+4754.355498422" watchObservedRunningTime="2026-11-03 21:20:24.222869136 +0000 UTC m=+4754.355997599"
Nov 03 21:20:30.211801 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:30.211765    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9"
Nov 03 21:20:34.204009 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:34.203980    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2"
Nov 03 21:20:34.204413 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:34.204047    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl"
Nov 03 21:20:34.381792 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:34.381755    4421 patch_prober.go:29] interesting pod/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": read tcp 10.134.0.2:46166->10.134.1.21:15021: read: connection reset by peer" start-of-body=
Nov 03 21:20:34.382568 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:34.381831    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv" podUID="d668277e-62c6-4f0e-ace6-a6b623f039f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": read tcp 10.134.0.2:46166->10.134.1.21:15021: read: connection reset by peer"
Nov 03 21:20:35.259453 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:35.259420    4421 generic.go:356] "Generic (PLEG): container finished" podID="d668277e-62c6-4f0e-ace6-a6b623f039f2" containerID="1d191b6e395a6c4fff1fd4355c23c21721cbe1d6809cb0cfbfd495264f5bc2b9" exitCode=0
Nov 03 21:20:35.259946 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:35.259493    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv" event={"ID":"d668277e-62c6-4f0e-ace6-a6b623f039f2","Type":"ContainerDied","Data":"1d191b6e395a6c4fff1fd4355c23c21721cbe1d6809cb0cfbfd495264f5bc2b9"}
Nov 03 21:20:35.259946 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:35.259874    4421 scope.go:122] "RemoveContainer" containerID="1d191b6e395a6c4fff1fd4355c23c21721cbe1d6809cb0cfbfd495264f5bc2b9"
Nov 03 21:20:35.261064 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:35.261037    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:20:35.261125 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:35.261112    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:20:35.261168 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:35.261152    4421 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 21:20:36.271415 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:36.271384    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv" event={"ID":"d668277e-62c6-4f0e-ace6-a6b623f039f2","Type":"ContainerStarted","Data":"d27de194b887b1cad8be2525995604935822293f13566e98659f1f987a4b3a76"}
Nov 03 21:20:37.070626 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:37.070002    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:37.070626 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:37.070127    4421 patch_prober.go:29] interesting pod/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Nov 03 21:20:37.070626 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:37.070170    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv" podUID="d668277e-62c6-4f0e-ace6-a6b623f039f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Nov 03 21:20:38.066124 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.066080    4421 patch_prober.go:29] interesting pod/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Nov 03 21:20:38.066531 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.066149    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv" podUID="d668277e-62c6-4f0e-ace6-a6b623f039f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Nov 03 21:20:38.613349 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.612088    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"]
Nov 03 21:20:38.616223 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.616165    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"
Nov 03 21:20:38.625046 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.624726    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"]
Nov 03 21:20:38.626065 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.626044    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-d9b592\"/\"default-dockercfg-sl9fw\""
Nov 03 21:20:38.629003 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.628982    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-d9b592\"/\"openshift-service-ca.crt\""
Nov 03 21:20:38.629246 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.629232    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-d9b592\"/\"kube-root-ca.crt\""
Nov 03 21:20:38.629512 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.629497    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-d9b592\"/\"simulator-single-main-d9b592-kserve-self-signed-certs\""
Nov 03 21:20:38.701645 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.701585    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-model-cache\") pod \"simulator-single-main-d9b592-kserve-c8d99d455-2phlx\" (UID: \"04bef5f4-355b-4c21-9333-dd98405661b5\") " pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"
Nov 03 21:20:38.701823 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.701712    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-home\") pod \"simulator-single-main-d9b592-kserve-c8d99d455-2phlx\" (UID: \"04bef5f4-355b-4c21-9333-dd98405661b5\") " pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"
Nov 03 21:20:38.701823 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.701744    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk6rv\" (UniqueName: \"kubernetes.io/projected/04bef5f4-355b-4c21-9333-dd98405661b5-kube-api-access-hk6rv\") pod \"simulator-single-main-d9b592-kserve-c8d99d455-2phlx\" (UID: \"04bef5f4-355b-4c21-9333-dd98405661b5\") " pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"
Nov 03 21:20:38.701823 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.701773    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-dshm\") pod \"simulator-single-main-d9b592-kserve-c8d99d455-2phlx\" (UID: \"04bef5f4-355b-4c21-9333-dd98405661b5\") " pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"
Nov 03 21:20:38.701933 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.701916    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04bef5f4-355b-4c21-9333-dd98405661b5-tls-certs\") pod \"simulator-single-main-d9b592-kserve-c8d99d455-2phlx\" (UID: \"04bef5f4-355b-4c21-9333-dd98405661b5\") " pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"
Nov 03 21:20:38.701975 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.701941    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-tmp-dir\") pod \"simulator-single-main-d9b592-kserve-c8d99d455-2phlx\" (UID: \"04bef5f4-355b-4c21-9333-dd98405661b5\") " pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"
Nov 03 21:20:38.803159 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.803122    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04bef5f4-355b-4c21-9333-dd98405661b5-tls-certs\") pod \"simulator-single-main-d9b592-kserve-c8d99d455-2phlx\" (UID: \"04bef5f4-355b-4c21-9333-dd98405661b5\") " pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"
Nov 03 21:20:38.803347 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.803167    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-tmp-dir\") pod \"simulator-single-main-d9b592-kserve-c8d99d455-2phlx\" (UID: \"04bef5f4-355b-4c21-9333-dd98405661b5\") " pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"
Nov 03 21:20:38.803347 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.803228    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-model-cache\") pod \"simulator-single-main-d9b592-kserve-c8d99d455-2phlx\" (UID: \"04bef5f4-355b-4c21-9333-dd98405661b5\") " pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"
Nov 03 21:20:38.803347 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.803281    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-home\") pod \"simulator-single-main-d9b592-kserve-c8d99d455-2phlx\" (UID: \"04bef5f4-355b-4c21-9333-dd98405661b5\") " pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"
Nov 03 21:20:38.803347 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.803308    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk6rv\" (UniqueName: \"kubernetes.io/projected/04bef5f4-355b-4c21-9333-dd98405661b5-kube-api-access-hk6rv\") pod \"simulator-single-main-d9b592-kserve-c8d99d455-2phlx\" (UID: \"04bef5f4-355b-4c21-9333-dd98405661b5\") " pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"
Nov 03 21:20:38.803347 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.803341    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-dshm\") pod \"simulator-single-main-d9b592-kserve-c8d99d455-2phlx\" (UID: \"04bef5f4-355b-4c21-9333-dd98405661b5\") " pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"
Nov 03 21:20:38.803986 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.803962    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-home\") pod \"simulator-single-main-d9b592-kserve-c8d99d455-2phlx\" (UID: \"04bef5f4-355b-4c21-9333-dd98405661b5\") " pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"
Nov 03 21:20:38.804227 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.804209    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-model-cache\") pod \"simulator-single-main-d9b592-kserve-c8d99d455-2phlx\" (UID: \"04bef5f4-355b-4c21-9333-dd98405661b5\") " pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"
Nov 03 21:20:38.804477 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.804301    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-tmp-dir\") pod \"simulator-single-main-d9b592-kserve-c8d99d455-2phlx\" (UID: \"04bef5f4-355b-4c21-9333-dd98405661b5\") " pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"
Nov 03 21:20:38.805930 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.805913    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-dshm\") pod \"simulator-single-main-d9b592-kserve-c8d99d455-2phlx\" (UID: \"04bef5f4-355b-4c21-9333-dd98405661b5\") " pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"
Nov 03 21:20:38.806050 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.805991    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04bef5f4-355b-4c21-9333-dd98405661b5-tls-certs\") pod \"simulator-single-main-d9b592-kserve-c8d99d455-2phlx\" (UID: \"04bef5f4-355b-4c21-9333-dd98405661b5\") " pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"
Nov 03 21:20:38.815769 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.815746    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk6rv\" (UniqueName: \"kubernetes.io/projected/04bef5f4-355b-4c21-9333-dd98405661b5-kube-api-access-hk6rv\") pod \"simulator-single-main-d9b592-kserve-c8d99d455-2phlx\" (UID: \"04bef5f4-355b-4c21-9333-dd98405661b5\") " pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"
Nov 03 21:20:38.935129 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:38.935057    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"
Nov 03 21:20:39.070254 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:39.069754    4421 patch_prober.go:29] interesting pod/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Nov 03 21:20:39.070254 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:39.069829    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv" podUID="d668277e-62c6-4f0e-ace6-a6b623f039f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Nov 03 21:20:39.238867 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:39.238817    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"]
Nov 03 21:20:39.261211 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:20:39.261103    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04bef5f4_355b_4c21_9333_dd98405661b5.slice/crio-efd345a8331dfd72b74d0ccd7a866685da4f87e5ae52228851f16d55ed95f8d0 WatchSource:0}: Error finding container efd345a8331dfd72b74d0ccd7a866685da4f87e5ae52228851f16d55ed95f8d0: Status 404 returned error can't find the container with id efd345a8331dfd72b74d0ccd7a866685da4f87e5ae52228851f16d55ed95f8d0
Nov 03 21:20:39.291618 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:39.291228    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx" event={"ID":"04bef5f4-355b-4c21-9333-dd98405661b5","Type":"ContainerStarted","Data":"efd345a8331dfd72b74d0ccd7a866685da4f87e5ae52228851f16d55ed95f8d0"}
Nov 03 21:20:40.066056 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:40.066015    4421 patch_prober.go:29] interesting pod/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Nov 03 21:20:40.066273 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:40.066083    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv" podUID="d668277e-62c6-4f0e-ace6-a6b623f039f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Nov 03 21:20:40.297475 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:40.297437    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx" event={"ID":"04bef5f4-355b-4c21-9333-dd98405661b5","Type":"ContainerStarted","Data":"a5646e5a7f2b9d23a6c7dd305e6ec502f9d4dca8f9889fd3a78079bd32853c15"}
Nov 03 21:20:40.319834 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:40.319740    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx" podStartSLOduration=2.319726337 podStartE2EDuration="2.319726337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:20:40.319029774 +0000 UTC m=+4770.452158236" watchObservedRunningTime="2026-11-03 21:20:40.319726337 +0000 UTC m=+4770.452854798"
Nov 03 21:20:40.935390 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:40.935347    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"
Nov 03 21:20:41.066622 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:41.065251    4421 patch_prober.go:29] interesting pod/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Nov 03 21:20:41.066622 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:41.065317    4421 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv" podUID="d668277e-62c6-4f0e-ace6-a6b623f039f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Nov 03 21:20:42.083265 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:42.083224    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:42.084089 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:42.084063    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:42.084635 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:42.084589    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:20:48.935896 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:48.935866    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"
Nov 03 21:20:48.948755 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:48.948726    4421 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"
Nov 03 21:20:49.357926 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:20:49.357900    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"
Nov 03 21:20:49.628377 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:20:49.628292    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:21:19.629139 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:21:19.629062    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:21:25.266067 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:25.266032    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"]
Nov 03 21:21:26.537414 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:26.537378    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl"]
Nov 03 21:21:26.554433 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:26.554401    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2"]
Nov 03 21:21:27.596164 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:27.596128    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9"]
Nov 03 21:21:27.596898 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:27.596478    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9" podUID="d0ce49d8-03d5-4881-bb0f-324349505550" containerName="maas-api" containerID="cri-o://2d6ddd3c966fd7e2364c2eef9609e3f8940b70f4159f233d22969f3e9ca07e7a" gracePeriod=30
Nov 03 21:21:27.872323 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:27.872300    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9"
Nov 03 21:21:28.012754 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:28.012726    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0ce49d8-03d5-4881-bb0f-324349505550-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0ce49d8-03d5-4881-bb0f-324349505550-metrics-certs\") pod \"d0ce49d8-03d5-4881-bb0f-324349505550\" (UID: \"d0ce49d8-03d5-4881-bb0f-324349505550\") "
Nov 03 21:21:28.012966 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:28.012767    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0ce49d8-03d5-4881-bb0f-324349505550-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0ce49d8-03d5-4881-bb0f-324349505550-openshift-service-ca\") pod \"d0ce49d8-03d5-4881-bb0f-324349505550\" (UID: \"d0ce49d8-03d5-4881-bb0f-324349505550\") "
Nov 03 21:21:28.012966 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:28.012816    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0ce49d8-03d5-4881-bb0f-324349505550-kube-api-access-rgvpd\" (UniqueName: \"kubernetes.io/projected/d0ce49d8-03d5-4881-bb0f-324349505550-kube-api-access-rgvpd\") pod \"d0ce49d8-03d5-4881-bb0f-324349505550\" (UID: \"d0ce49d8-03d5-4881-bb0f-324349505550\") "
Nov 03 21:21:28.012966 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:28.012855    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0ce49d8-03d5-4881-bb0f-324349505550-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0ce49d8-03d5-4881-bb0f-324349505550-maas-api-tls\") pod \"d0ce49d8-03d5-4881-bb0f-324349505550\" (UID: \"d0ce49d8-03d5-4881-bb0f-324349505550\") "
Nov 03 21:21:28.013151 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:28.013124    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0ce49d8-03d5-4881-bb0f-324349505550-openshift-service-ca" pod "d0ce49d8-03d5-4881-bb0f-324349505550" (UID: "d0ce49d8-03d5-4881-bb0f-324349505550"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:21:28.013294 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:28.013264    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0ce49d8-03d5-4881-bb0f-324349505550-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:28.015212 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:28.015185    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0ce49d8-03d5-4881-bb0f-324349505550-kube-api-access-rgvpd" pod "d0ce49d8-03d5-4881-bb0f-324349505550" (UID: "d0ce49d8-03d5-4881-bb0f-324349505550"). InnerVolumeSpecName "kube-api-access-rgvpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:21:28.015688 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:28.015661    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0ce49d8-03d5-4881-bb0f-324349505550-metrics-certs" pod "d0ce49d8-03d5-4881-bb0f-324349505550" (UID: "d0ce49d8-03d5-4881-bb0f-324349505550"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:21:28.015688 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:28.015682    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0ce49d8-03d5-4881-bb0f-324349505550-maas-api-tls" pod "d0ce49d8-03d5-4881-bb0f-324349505550" (UID: "d0ce49d8-03d5-4881-bb0f-324349505550"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:21:28.114882 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:28.114783    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0ce49d8-03d5-4881-bb0f-324349505550-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:28.114882 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:28.114814    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgvpd\" (UniqueName: \"kubernetes.io/projected/d0ce49d8-03d5-4881-bb0f-324349505550-kube-api-access-rgvpd\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:28.114882 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:28.114824    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0ce49d8-03d5-4881-bb0f-324349505550-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:28.530463 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:28.530426    4421 generic.go:356] "Generic (PLEG): container finished" podID="d0ce49d8-03d5-4881-bb0f-324349505550" containerID="2d6ddd3c966fd7e2364c2eef9609e3f8940b70f4159f233d22969f3e9ca07e7a" exitCode=0
Nov 03 21:21:28.530691 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:28.530475    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9"
Nov 03 21:21:28.530691 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:28.530515    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9" event={"ID":"d0ce49d8-03d5-4881-bb0f-324349505550","Type":"ContainerDied","Data":"2d6ddd3c966fd7e2364c2eef9609e3f8940b70f4159f233d22969f3e9ca07e7a"}
Nov 03 21:21:28.530691 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:28.530575    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9" event={"ID":"d0ce49d8-03d5-4881-bb0f-324349505550","Type":"ContainerDied","Data":"52b9e881a501d2c4b52a4a43d84bb57a2038dcf316a9478fe49f6d5620041870"}
Nov 03 21:21:28.530691 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:28.530620    4421 scope.go:122] "RemoveContainer" containerID="2d6ddd3c966fd7e2364c2eef9609e3f8940b70f4159f233d22969f3e9ca07e7a"
Nov 03 21:21:28.539510 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:28.539495    4421 scope.go:122] "RemoveContainer" containerID="2d6ddd3c966fd7e2364c2eef9609e3f8940b70f4159f233d22969f3e9ca07e7a"
Nov 03 21:21:28.539777 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:21:28.539758    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d6ddd3c966fd7e2364c2eef9609e3f8940b70f4159f233d22969f3e9ca07e7a\": container with ID starting with 2d6ddd3c966fd7e2364c2eef9609e3f8940b70f4159f233d22969f3e9ca07e7a not found: ID does not exist" containerID="2d6ddd3c966fd7e2364c2eef9609e3f8940b70f4159f233d22969f3e9ca07e7a"
Nov 03 21:21:28.539838 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:28.539786    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d6ddd3c966fd7e2364c2eef9609e3f8940b70f4159f233d22969f3e9ca07e7a"} err="failed to get container status \"2d6ddd3c966fd7e2364c2eef9609e3f8940b70f4159f233d22969f3e9ca07e7a\": rpc error: code = NotFound desc = could not find container \"2d6ddd3c966fd7e2364c2eef9609e3f8940b70f4159f233d22969f3e9ca07e7a\": container with ID starting with 2d6ddd3c966fd7e2364c2eef9609e3f8940b70f4159f233d22969f3e9ca07e7a not found: ID does not exist"
Nov 03 21:21:28.553318 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:28.553292    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9"]
Nov 03 21:21:28.557852 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:28.557831    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-dhqh9"]
Nov 03 21:21:30.014560 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:30.014524    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0ce49d8-03d5-4881-bb0f-324349505550" path="/var/lib/kubelet/pods/d0ce49d8-03d5-4881-bb0f-324349505550/volumes"
Nov 03 21:21:31.538589 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:31.538545    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl" podUID="bcb2f012-348e-4395-97f3-eed7277d44fd" containerName="payload-processing" containerID="cri-o://3c4b239946fd41f5772f7847a0c47b02dd0b88bc3e405cb4bb2259c0b37ff9a2" gracePeriod=25
Nov 03 21:21:31.554944 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:31.554908    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2" podUID="a43cea43-6f73-40c7-b16c-a3502238ab28" containerName="payload-pre-processing" containerID="cri-o://cd40e1e48e2c8f59ba6efea2d238015b36dc5dacb057e2b996b81fee90e7d5fd" gracePeriod=25
Nov 03 21:21:34.202985 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:34.202940    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2" podUID="a43cea43-6f73-40c7-b16c-a3502238ab28" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:21:34.203398 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:34.202955    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl" podUID="bcb2f012-348e-4395-97f3-eed7277d44fd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:21:40.286935 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.286866    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx" podUID="04bef5f4-355b-4c21-9333-dd98405661b5" containerName="main" containerID="cri-o://a5646e5a7f2b9d23a6c7dd305e6ec502f9d4dca8f9889fd3a78079bd32853c15" gracePeriod=45
Nov 03 21:21:40.343843 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.343811    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx" podUID="04bef5f4-355b-4c21-9333-dd98405661b5" containerName="main" probeResult="failure" output="Get \"https://10.134.1.25:8000/ready\": dial tcp 10.134.1.25:8000: connect: connection refused"
Nov 03 21:21:40.535887 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.535862    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"
Nov 03 21:21:40.593323 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.593224    4421 generic.go:356] "Generic (PLEG): container finished" podID="04bef5f4-355b-4c21-9333-dd98405661b5" containerID="a5646e5a7f2b9d23a6c7dd305e6ec502f9d4dca8f9889fd3a78079bd32853c15" exitCode=0
Nov 03 21:21:40.593323 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.593288    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"
Nov 03 21:21:40.593323 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.593311    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx" event={"ID":"04bef5f4-355b-4c21-9333-dd98405661b5","Type":"ContainerDied","Data":"a5646e5a7f2b9d23a6c7dd305e6ec502f9d4dca8f9889fd3a78079bd32853c15"}
Nov 03 21:21:40.593631 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.593360    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx" event={"ID":"04bef5f4-355b-4c21-9333-dd98405661b5","Type":"ContainerDied","Data":"efd345a8331dfd72b74d0ccd7a866685da4f87e5ae52228851f16d55ed95f8d0"}
Nov 03 21:21:40.593631 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.593384    4421 scope.go:122] "RemoveContainer" containerID="a5646e5a7f2b9d23a6c7dd305e6ec502f9d4dca8f9889fd3a78079bd32853c15"
Nov 03 21:21:40.603532 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.603508    4421 scope.go:122] "RemoveContainer" containerID="a5646e5a7f2b9d23a6c7dd305e6ec502f9d4dca8f9889fd3a78079bd32853c15"
Nov 03 21:21:40.603800 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:21:40.603781    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5646e5a7f2b9d23a6c7dd305e6ec502f9d4dca8f9889fd3a78079bd32853c15\": container with ID starting with a5646e5a7f2b9d23a6c7dd305e6ec502f9d4dca8f9889fd3a78079bd32853c15 not found: ID does not exist" containerID="a5646e5a7f2b9d23a6c7dd305e6ec502f9d4dca8f9889fd3a78079bd32853c15"
Nov 03 21:21:40.603909 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.603809    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5646e5a7f2b9d23a6c7dd305e6ec502f9d4dca8f9889fd3a78079bd32853c15"} err="failed to get container status \"a5646e5a7f2b9d23a6c7dd305e6ec502f9d4dca8f9889fd3a78079bd32853c15\": rpc error: code = NotFound desc = could not find container \"a5646e5a7f2b9d23a6c7dd305e6ec502f9d4dca8f9889fd3a78079bd32853c15\": container with ID starting with a5646e5a7f2b9d23a6c7dd305e6ec502f9d4dca8f9889fd3a78079bd32853c15 not found: ID does not exist"
Nov 03 21:21:40.639875 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.639847    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-home\" (UniqueName: \"kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-home\") pod \"04bef5f4-355b-4c21-9333-dd98405661b5\" (UID: \"04bef5f4-355b-4c21-9333-dd98405661b5\") "
Nov 03 21:21:40.639875 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.639880    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04bef5f4-355b-4c21-9333-dd98405661b5-tls-certs\" (UniqueName: \"kubernetes.io/secret/04bef5f4-355b-4c21-9333-dd98405661b5-tls-certs\") pod \"04bef5f4-355b-4c21-9333-dd98405661b5\" (UID: \"04bef5f4-355b-4c21-9333-dd98405661b5\") "
Nov 03 21:21:40.640141 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.639925    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-tmp-dir\") pod \"04bef5f4-355b-4c21-9333-dd98405661b5\" (UID: \"04bef5f4-355b-4c21-9333-dd98405661b5\") "
Nov 03 21:21:40.640141 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.639965    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04bef5f4-355b-4c21-9333-dd98405661b5-kube-api-access-hk6rv\" (UniqueName: \"kubernetes.io/projected/04bef5f4-355b-4c21-9333-dd98405661b5-kube-api-access-hk6rv\") pod \"04bef5f4-355b-4c21-9333-dd98405661b5\" (UID: \"04bef5f4-355b-4c21-9333-dd98405661b5\") "
Nov 03 21:21:40.640141 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.640031    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-model-cache\") pod \"04bef5f4-355b-4c21-9333-dd98405661b5\" (UID: \"04bef5f4-355b-4c21-9333-dd98405661b5\") "
Nov 03 21:21:40.640324 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.640143    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-dshm\") pod \"04bef5f4-355b-4c21-9333-dd98405661b5\" (UID: \"04bef5f4-355b-4c21-9333-dd98405661b5\") "
Nov 03 21:21:40.640324 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.640227    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-tmp-dir" pod "04bef5f4-355b-4c21-9333-dd98405661b5" (UID: "04bef5f4-355b-4c21-9333-dd98405661b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:21:40.640449 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.640371    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-model-cache" pod "04bef5f4-355b-4c21-9333-dd98405661b5" (UID: "04bef5f4-355b-4c21-9333-dd98405661b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:21:40.640449 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.640401    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-home" pod "04bef5f4-355b-4c21-9333-dd98405661b5" (UID: "04bef5f4-355b-4c21-9333-dd98405661b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:21:40.640561 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.640503    4421 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-model-cache\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:40.640561 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.640524    4421 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-home\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:40.640561 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.640537    4421 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-tmp-dir\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:40.642522 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.642492    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-dshm" pod "04bef5f4-355b-4c21-9333-dd98405661b5" (UID: "04bef5f4-355b-4c21-9333-dd98405661b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:21:40.642522 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.642494    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04bef5f4-355b-4c21-9333-dd98405661b5-kube-api-access-hk6rv" pod "04bef5f4-355b-4c21-9333-dd98405661b5" (UID: "04bef5f4-355b-4c21-9333-dd98405661b5"). InnerVolumeSpecName "kube-api-access-hk6rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:21:40.642698 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.642652    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bef5f4-355b-4c21-9333-dd98405661b5-tls-certs" pod "04bef5f4-355b-4c21-9333-dd98405661b5" (UID: "04bef5f4-355b-4c21-9333-dd98405661b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:21:40.741458 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.741429    4421 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04bef5f4-355b-4c21-9333-dd98405661b5-dshm\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:40.741458 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.741456    4421 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04bef5f4-355b-4c21-9333-dd98405661b5-tls-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:40.741458 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.741466    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk6rv\" (UniqueName: \"kubernetes.io/projected/04bef5f4-355b-4c21-9333-dd98405661b5-kube-api-access-hk6rv\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:40.917383 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.917352    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"]
Nov 03 21:21:40.922013 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:40.921989    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-d9b592/simulator-single-main-d9b592-kserve-c8d99d455-2phlx"]
Nov 03 21:21:42.015582 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:42.015545    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04bef5f4-355b-4c21-9333-dd98405661b5" path="/var/lib/kubelet/pods/04bef5f4-355b-4c21-9333-dd98405661b5/volumes"
Nov 03 21:21:44.203064 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:44.203026    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl" podUID="bcb2f012-348e-4395-97f3-eed7277d44fd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:21:44.203456 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:44.203093    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2" podUID="a43cea43-6f73-40c7-b16c-a3502238ab28" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 21:21:49.627900 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:21:49.627865    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:21:51.842075 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:51.842053    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl"
Nov 03 21:21:51.844974 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:51.844955    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2"
Nov 03 21:21:51.947959 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:51.947924    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a43cea43-6f73-40c7-b16c-a3502238ab28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a43cea43-6f73-40c7-b16c-a3502238ab28-plugins-config-volume\") pod \"a43cea43-6f73-40c7-b16c-a3502238ab28\" (UID: \"a43cea43-6f73-40c7-b16c-a3502238ab28\") "
Nov 03 21:21:51.948164 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:51.948001    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcb2f012-348e-4395-97f3-eed7277d44fd-kube-api-access-s7bqz\" (UniqueName: \"kubernetes.io/projected/bcb2f012-348e-4395-97f3-eed7277d44fd-kube-api-access-s7bqz\") pod \"bcb2f012-348e-4395-97f3-eed7277d44fd\" (UID: \"bcb2f012-348e-4395-97f3-eed7277d44fd\") "
Nov 03 21:21:51.948164 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:51.948032    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a43cea43-6f73-40c7-b16c-a3502238ab28-kube-api-access-4hwdb\" (UniqueName: \"kubernetes.io/projected/a43cea43-6f73-40c7-b16c-a3502238ab28-kube-api-access-4hwdb\") pod \"a43cea43-6f73-40c7-b16c-a3502238ab28\" (UID: \"a43cea43-6f73-40c7-b16c-a3502238ab28\") "
Nov 03 21:21:51.948164 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:51.948115    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcb2f012-348e-4395-97f3-eed7277d44fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcb2f012-348e-4395-97f3-eed7277d44fd-plugins-config-volume\") pod \"bcb2f012-348e-4395-97f3-eed7277d44fd\" (UID: \"bcb2f012-348e-4395-97f3-eed7277d44fd\") "
Nov 03 21:21:51.948447 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:51.948415    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a43cea43-6f73-40c7-b16c-a3502238ab28-plugins-config-volume" pod "a43cea43-6f73-40c7-b16c-a3502238ab28" (UID: "a43cea43-6f73-40c7-b16c-a3502238ab28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:21:51.948565 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:51.948507    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcb2f012-348e-4395-97f3-eed7277d44fd-plugins-config-volume" pod "bcb2f012-348e-4395-97f3-eed7277d44fd" (UID: "bcb2f012-348e-4395-97f3-eed7277d44fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:21:51.950351 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:51.950332    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb2f012-348e-4395-97f3-eed7277d44fd-kube-api-access-s7bqz" pod "bcb2f012-348e-4395-97f3-eed7277d44fd" (UID: "bcb2f012-348e-4395-97f3-eed7277d44fd"). InnerVolumeSpecName "kube-api-access-s7bqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:21:51.950452 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:51.950434    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a43cea43-6f73-40c7-b16c-a3502238ab28-kube-api-access-4hwdb" pod "a43cea43-6f73-40c7-b16c-a3502238ab28" (UID: "a43cea43-6f73-40c7-b16c-a3502238ab28"). InnerVolumeSpecName "kube-api-access-4hwdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:21:52.049777 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:52.049750    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7bqz\" (UniqueName: \"kubernetes.io/projected/bcb2f012-348e-4395-97f3-eed7277d44fd-kube-api-access-s7bqz\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:52.049777 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:52.049776    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hwdb\" (UniqueName: \"kubernetes.io/projected/a43cea43-6f73-40c7-b16c-a3502238ab28-kube-api-access-4hwdb\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:52.049949 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:52.049785    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcb2f012-348e-4395-97f3-eed7277d44fd-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:52.049949 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:52.049794    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a43cea43-6f73-40c7-b16c-a3502238ab28-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:21:52.649651 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:52.649592    4421 generic.go:356] "Generic (PLEG): container finished" podID="bcb2f012-348e-4395-97f3-eed7277d44fd" containerID="3c4b239946fd41f5772f7847a0c47b02dd0b88bc3e405cb4bb2259c0b37ff9a2" exitCode=0
Nov 03 21:21:52.649651 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:52.649654    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl"
Nov 03 21:21:52.649960 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:52.649703    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl" event={"ID":"bcb2f012-348e-4395-97f3-eed7277d44fd","Type":"ContainerDied","Data":"3c4b239946fd41f5772f7847a0c47b02dd0b88bc3e405cb4bb2259c0b37ff9a2"}
Nov 03 21:21:52.649960 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:52.649766    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl" event={"ID":"bcb2f012-348e-4395-97f3-eed7277d44fd","Type":"ContainerDied","Data":"89869cd53fd16cf7b6e30e6c1a4cca710fcd58994b5c022f53ee1c31207b108a"}
Nov 03 21:21:52.649960 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:52.649797    4421 scope.go:122] "RemoveContainer" containerID="3c4b239946fd41f5772f7847a0c47b02dd0b88bc3e405cb4bb2259c0b37ff9a2"
Nov 03 21:21:52.651132 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:52.651107    4421 generic.go:356] "Generic (PLEG): container finished" podID="a43cea43-6f73-40c7-b16c-a3502238ab28" containerID="cd40e1e48e2c8f59ba6efea2d238015b36dc5dacb057e2b996b81fee90e7d5fd" exitCode=0
Nov 03 21:21:52.651266 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:52.651154    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2" event={"ID":"a43cea43-6f73-40c7-b16c-a3502238ab28","Type":"ContainerDied","Data":"cd40e1e48e2c8f59ba6efea2d238015b36dc5dacb057e2b996b81fee90e7d5fd"}
Nov 03 21:21:52.651266 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:52.651193    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2" event={"ID":"a43cea43-6f73-40c7-b16c-a3502238ab28","Type":"ContainerDied","Data":"85f6c7220db19710608b86e9b07e1c653e528b5b2bf3ea499e38802786d03c35"}
Nov 03 21:21:52.651266 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:52.651156    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2"
Nov 03 21:21:52.658543 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:52.658529    4421 scope.go:122] "RemoveContainer" containerID="3c4b239946fd41f5772f7847a0c47b02dd0b88bc3e405cb4bb2259c0b37ff9a2"
Nov 03 21:21:52.658790 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:21:52.658772    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c4b239946fd41f5772f7847a0c47b02dd0b88bc3e405cb4bb2259c0b37ff9a2\": container with ID starting with 3c4b239946fd41f5772f7847a0c47b02dd0b88bc3e405cb4bb2259c0b37ff9a2 not found: ID does not exist" containerID="3c4b239946fd41f5772f7847a0c47b02dd0b88bc3e405cb4bb2259c0b37ff9a2"
Nov 03 21:21:52.658838 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:52.658805    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c4b239946fd41f5772f7847a0c47b02dd0b88bc3e405cb4bb2259c0b37ff9a2"} err="failed to get container status \"3c4b239946fd41f5772f7847a0c47b02dd0b88bc3e405cb4bb2259c0b37ff9a2\": rpc error: code = NotFound desc = could not find container \"3c4b239946fd41f5772f7847a0c47b02dd0b88bc3e405cb4bb2259c0b37ff9a2\": container with ID starting with 3c4b239946fd41f5772f7847a0c47b02dd0b88bc3e405cb4bb2259c0b37ff9a2 not found: ID does not exist"
Nov 03 21:21:52.658838 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:52.658823    4421 scope.go:122] "RemoveContainer" containerID="cd40e1e48e2c8f59ba6efea2d238015b36dc5dacb057e2b996b81fee90e7d5fd"
Nov 03 21:21:52.667240 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:52.667090    4421 scope.go:122] "RemoveContainer" containerID="cd40e1e48e2c8f59ba6efea2d238015b36dc5dacb057e2b996b81fee90e7d5fd"
Nov 03 21:21:52.667373 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:21:52.667351    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd40e1e48e2c8f59ba6efea2d238015b36dc5dacb057e2b996b81fee90e7d5fd\": container with ID starting with cd40e1e48e2c8f59ba6efea2d238015b36dc5dacb057e2b996b81fee90e7d5fd not found: ID does not exist" containerID="cd40e1e48e2c8f59ba6efea2d238015b36dc5dacb057e2b996b81fee90e7d5fd"
Nov 03 21:21:52.667418 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:52.667381    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd40e1e48e2c8f59ba6efea2d238015b36dc5dacb057e2b996b81fee90e7d5fd"} err="failed to get container status \"cd40e1e48e2c8f59ba6efea2d238015b36dc5dacb057e2b996b81fee90e7d5fd\": rpc error: code = NotFound desc = could not find container \"cd40e1e48e2c8f59ba6efea2d238015b36dc5dacb057e2b996b81fee90e7d5fd\": container with ID starting with cd40e1e48e2c8f59ba6efea2d238015b36dc5dacb057e2b996b81fee90e7d5fd not found: ID does not exist"
Nov 03 21:21:52.668162 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:52.668145    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl"]
Nov 03 21:21:52.673202 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:52.673182    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877px9vl"]
Nov 03 21:21:52.683456 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:52.683436    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2"]
Nov 03 21:21:52.687590 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:52.687568    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc965dz2"]
Nov 03 21:21:54.014447 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:54.014414    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a43cea43-6f73-40c7-b16c-a3502238ab28" path="/var/lib/kubelet/pods/a43cea43-6f73-40c7-b16c-a3502238ab28/volumes"
Nov 03 21:21:54.014850 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:54.014781    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcb2f012-348e-4395-97f3-eed7277d44fd" path="/var/lib/kubelet/pods/bcb2f012-348e-4395-97f3-eed7277d44fd/volumes"
Nov 03 21:21:58.942740 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:58.942708    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6"]
Nov 03 21:21:58.943681 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:58.943314    4421 cpu_manager.go:414] "containerMap: removing container" podUID="bcb2f012-348e-4395-97f3-eed7277d44fd" containerName="payload-processing"
Nov 03 21:21:58.943681 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:58.943336    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcb2f012-348e-4395-97f3-eed7277d44fd" containerName="payload-processing"
Nov 03 21:21:58.943681 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:58.943353    4421 cpu_manager.go:414] "containerMap: removing container" podUID="a43cea43-6f73-40c7-b16c-a3502238ab28" containerName="payload-pre-processing"
Nov 03 21:21:58.943681 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:58.943360    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a43cea43-6f73-40c7-b16c-a3502238ab28" containerName="payload-pre-processing"
Nov 03 21:21:58.943681 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:58.943375    4421 cpu_manager.go:414] "containerMap: removing container" podUID="d0ce49d8-03d5-4881-bb0f-324349505550" containerName="maas-api"
Nov 03 21:21:58.943681 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:58.943381    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0ce49d8-03d5-4881-bb0f-324349505550" containerName="maas-api"
Nov 03 21:21:58.943681 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:58.943393    4421 cpu_manager.go:414] "containerMap: removing container" podUID="04bef5f4-355b-4c21-9333-dd98405661b5" containerName="main"
Nov 03 21:21:58.943681 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:58.943400    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04bef5f4-355b-4c21-9333-dd98405661b5" containerName="main"
Nov 03 21:21:58.943681 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:58.943511    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcb2f012-348e-4395-97f3-eed7277d44fd" containerName="payload-processing"
Nov 03 21:21:58.943681 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:58.943526    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a43cea43-6f73-40c7-b16c-a3502238ab28" containerName="payload-pre-processing"
Nov 03 21:21:58.943681 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:58.943537    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0ce49d8-03d5-4881-bb0f-324349505550" containerName="maas-api"
Nov 03 21:21:58.943681 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:58.943546    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="04bef5f4-355b-4c21-9333-dd98405661b5" containerName="main"
Nov 03 21:21:58.948629 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:58.948583    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6"
Nov 03 21:21:58.956846 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:58.956782    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-d9b592-serving-cert\""
Nov 03 21:21:58.962214 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:58.962179    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6"]
Nov 03 21:21:59.016820 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.016788    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aed4032-16a6-4a43-b278-50e496299a92-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6\" (UID: \"4aed4032-16a6-4a43-b278-50e496299a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6"
Nov 03 21:21:59.017026 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.016850    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aed4032-16a6-4a43-b278-50e496299a92-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6\" (UID: \"4aed4032-16a6-4a43-b278-50e496299a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6"
Nov 03 21:21:59.017104 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.017041    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2294b\" (UniqueName: \"kubernetes.io/projected/4aed4032-16a6-4a43-b278-50e496299a92-kube-api-access-2294b\") pod \"maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6\" (UID: \"4aed4032-16a6-4a43-b278-50e496299a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6"
Nov 03 21:21:59.017179 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.017158    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aed4032-16a6-4a43-b278-50e496299a92-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6\" (UID: \"4aed4032-16a6-4a43-b278-50e496299a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6"
Nov 03 21:21:59.118008 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.117975    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aed4032-16a6-4a43-b278-50e496299a92-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6\" (UID: \"4aed4032-16a6-4a43-b278-50e496299a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6"
Nov 03 21:21:59.118204 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.118063    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aed4032-16a6-4a43-b278-50e496299a92-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6\" (UID: \"4aed4032-16a6-4a43-b278-50e496299a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6"
Nov 03 21:21:59.118204 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.118103    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aed4032-16a6-4a43-b278-50e496299a92-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6\" (UID: \"4aed4032-16a6-4a43-b278-50e496299a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6"
Nov 03 21:21:59.118204 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.118167    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2294b\" (UniqueName: \"kubernetes.io/projected/4aed4032-16a6-4a43-b278-50e496299a92-kube-api-access-2294b\") pod \"maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6\" (UID: \"4aed4032-16a6-4a43-b278-50e496299a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6"
Nov 03 21:21:59.118768 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.118735    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aed4032-16a6-4a43-b278-50e496299a92-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6\" (UID: \"4aed4032-16a6-4a43-b278-50e496299a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6"
Nov 03 21:21:59.120462 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.120434    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aed4032-16a6-4a43-b278-50e496299a92-metrics-certs\") pod \"maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6\" (UID: \"4aed4032-16a6-4a43-b278-50e496299a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6"
Nov 03 21:21:59.120697 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.120679    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aed4032-16a6-4a43-b278-50e496299a92-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6\" (UID: \"4aed4032-16a6-4a43-b278-50e496299a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6"
Nov 03 21:21:59.147836 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.147811    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2294b\" (UniqueName: \"kubernetes.io/projected/4aed4032-16a6-4a43-b278-50e496299a92-kube-api-access-2294b\") pod \"maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6\" (UID: \"4aed4032-16a6-4a43-b278-50e496299a92\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6"
Nov 03 21:21:59.164730 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.164702    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v"]
Nov 03 21:21:59.168591 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.168571    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg"]
Nov 03 21:21:59.168737 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.168696    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v"
Nov 03 21:21:59.171906 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.171890    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg"
Nov 03 21:21:59.177067 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.177040    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-d9b592\""
Nov 03 21:21:59.179581 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.179552    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v"]
Nov 03 21:21:59.195547 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.195487    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg"]
Nov 03 21:21:59.266352 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.266313    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6"
Nov 03 21:21:59.336801 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.321253    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj44b\" (UniqueName: \"kubernetes.io/projected/a03cb02c-8dcc-4a83-aba3-6c1309c86212-kube-api-access-qj44b\") pod \"payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v\" (UID: \"a03cb02c-8dcc-4a83-aba3-6c1309c86212\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v"
Nov 03 21:21:59.336801 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.324410    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxfk2\" (UniqueName: \"kubernetes.io/projected/24eef40b-2c73-43c7-8357-1f03e193f5f6-kube-api-access-dxfk2\") pod \"payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg\" (UID: \"24eef40b-2c73-43c7-8357-1f03e193f5f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg"
Nov 03 21:21:59.336801 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.324628    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24eef40b-2c73-43c7-8357-1f03e193f5f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg\" (UID: \"24eef40b-2c73-43c7-8357-1f03e193f5f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg"
Nov 03 21:21:59.336801 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.324818    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a03cb02c-8dcc-4a83-aba3-6c1309c86212-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v\" (UID: \"a03cb02c-8dcc-4a83-aba3-6c1309c86212\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v"
Nov 03 21:21:59.425463 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.425420    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a03cb02c-8dcc-4a83-aba3-6c1309c86212-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v\" (UID: \"a03cb02c-8dcc-4a83-aba3-6c1309c86212\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v"
Nov 03 21:21:59.425685 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.425513    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj44b\" (UniqueName: \"kubernetes.io/projected/a03cb02c-8dcc-4a83-aba3-6c1309c86212-kube-api-access-qj44b\") pod \"payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v\" (UID: \"a03cb02c-8dcc-4a83-aba3-6c1309c86212\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v"
Nov 03 21:21:59.425685 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.425546    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxfk2\" (UniqueName: \"kubernetes.io/projected/24eef40b-2c73-43c7-8357-1f03e193f5f6-kube-api-access-dxfk2\") pod \"payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg\" (UID: \"24eef40b-2c73-43c7-8357-1f03e193f5f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg"
Nov 03 21:21:59.425685 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.425583    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24eef40b-2c73-43c7-8357-1f03e193f5f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg\" (UID: \"24eef40b-2c73-43c7-8357-1f03e193f5f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg"
Nov 03 21:21:59.426225 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.426204    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a03cb02c-8dcc-4a83-aba3-6c1309c86212-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v\" (UID: \"a03cb02c-8dcc-4a83-aba3-6c1309c86212\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v"
Nov 03 21:21:59.426356 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.426339    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24eef40b-2c73-43c7-8357-1f03e193f5f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg\" (UID: \"24eef40b-2c73-43c7-8357-1f03e193f5f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg"
Nov 03 21:21:59.438458 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.438428    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj44b\" (UniqueName: \"kubernetes.io/projected/a03cb02c-8dcc-4a83-aba3-6c1309c86212-kube-api-access-qj44b\") pod \"payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v\" (UID: \"a03cb02c-8dcc-4a83-aba3-6c1309c86212\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v"
Nov 03 21:21:59.438554 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.438496    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxfk2\" (UniqueName: \"kubernetes.io/projected/24eef40b-2c73-43c7-8357-1f03e193f5f6-kube-api-access-dxfk2\") pod \"payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg\" (UID: \"24eef40b-2c73-43c7-8357-1f03e193f5f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg"
Nov 03 21:21:59.485915 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.485837    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v"
Nov 03 21:21:59.494296 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.494271    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg"
Nov 03 21:21:59.573057 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.570089    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6"]
Nov 03 21:21:59.690059 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.690026    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6" event={"ID":"4aed4032-16a6-4a43-b278-50e496299a92","Type":"ContainerStarted","Data":"006340b8b0c66cbd2b139dcb4d06682467e113180743b111d84c2bb113894cdc"}
Nov 03 21:21:59.772777 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.772649    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v"]
Nov 03 21:21:59.777092 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:21:59.775549    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg"]
Nov 03 21:21:59.777092 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:21:59.776149    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda03cb02c_8dcc_4a83_aba3_6c1309c86212.slice/crio-2332d7d1d6e191257ace9b24d37914b5ca89d09bf2a1b10b6134f93a9ffa8e5f WatchSource:0}: Error finding container 2332d7d1d6e191257ace9b24d37914b5ca89d09bf2a1b10b6134f93a9ffa8e5f: Status 404 returned error can't find the container with id 2332d7d1d6e191257ace9b24d37914b5ca89d09bf2a1b10b6134f93a9ffa8e5f
Nov 03 21:21:59.779320 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:21:59.779288    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24eef40b_2c73_43c7_8357_1f03e193f5f6.slice/crio-fc027dc434169df1a407e383f4e4fc8537f8ac7432a138b283dda57f2b9940ad WatchSource:0}: Error finding container fc027dc434169df1a407e383f4e4fc8537f8ac7432a138b283dda57f2b9940ad: Status 404 returned error can't find the container with id fc027dc434169df1a407e383f4e4fc8537f8ac7432a138b283dda57f2b9940ad
Nov 03 21:22:00.696478 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:00.696446    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v" event={"ID":"a03cb02c-8dcc-4a83-aba3-6c1309c86212","Type":"ContainerStarted","Data":"66684d68ac885f794b9d3e704977ed41d6cc3b1a18143135fa11194b3a1a2fb7"}
Nov 03 21:22:00.696967 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:00.696503    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v" event={"ID":"a03cb02c-8dcc-4a83-aba3-6c1309c86212","Type":"ContainerStarted","Data":"2332d7d1d6e191257ace9b24d37914b5ca89d09bf2a1b10b6134f93a9ffa8e5f"}
Nov 03 21:22:00.696967 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:00.696625    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v"
Nov 03 21:22:00.698047 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:00.698025    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6" event={"ID":"4aed4032-16a6-4a43-b278-50e496299a92","Type":"ContainerStarted","Data":"729529ec01f6a2b6b2cc9287b29540257cd54820464e4f005a6c80c9fae54cb0"}
Nov 03 21:22:00.698176 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:00.698146    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6"
Nov 03 21:22:00.699651 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:00.699628    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg" event={"ID":"24eef40b-2c73-43c7-8357-1f03e193f5f6","Type":"ContainerStarted","Data":"ad0aec55054bd99c0e78da0ce51408e8af137ace6eb582df7dc4bafdc8f12649"}
Nov 03 21:22:00.699780 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:00.699673    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg" event={"ID":"24eef40b-2c73-43c7-8357-1f03e193f5f6","Type":"ContainerStarted","Data":"fc027dc434169df1a407e383f4e4fc8537f8ac7432a138b283dda57f2b9940ad"}
Nov 03 21:22:00.699780 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:00.699708    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg"
Nov 03 21:22:00.717367 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:00.717310    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v" podStartSLOduration=1.717291045 podStartE2EDuration="1.717291045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:22:00.714180789 +0000 UTC m=+4850.847309252" watchObservedRunningTime="2026-11-03 21:22:00.717291045 +0000 UTC m=+4850.850419509"
Nov 03 21:22:00.735978 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:00.735928    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg" podStartSLOduration=1.7359155899999998 podStartE2EDuration="1.73591559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:22:00.733682863 +0000 UTC m=+4850.866811331" watchObservedRunningTime="2026-11-03 21:22:00.73591559 +0000 UTC m=+4850.869044051"
Nov 03 21:22:00.755283 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:00.755236    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6" podStartSLOduration=2.148198481 podStartE2EDuration="2.755222001s" totalImagesPullingTime="607.02352ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:21:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:22:00.753348178 +0000 UTC m=+4850.886476634" watchObservedRunningTime="2026-11-03 21:22:00.755222001 +0000 UTC m=+4850.888350487"
Nov 03 21:22:06.710246 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:06.710215    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6"
Nov 03 21:22:11.707159 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:11.707125    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg"
Nov 03 21:22:11.707592 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:11.707202    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v"
Nov 03 21:22:17.635372 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:17.635337    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v"]
Nov 03 21:22:17.652719 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:17.652684    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg"]
Nov 03 21:22:18.135705 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:18.135672    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6"]
Nov 03 21:22:18.135949 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:18.135917    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6" podUID="4aed4032-16a6-4a43-b278-50e496299a92" containerName="maas-api" containerID="cri-o://729529ec01f6a2b6b2cc9287b29540257cd54820464e4f005a6c80c9fae54cb0" gracePeriod=30
Nov 03 21:22:18.412040 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:18.412013    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6"
Nov 03 21:22:18.423749 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:18.423720    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aed4032-16a6-4a43-b278-50e496299a92-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aed4032-16a6-4a43-b278-50e496299a92-openshift-service-ca\") pod \"4aed4032-16a6-4a43-b278-50e496299a92\" (UID: \"4aed4032-16a6-4a43-b278-50e496299a92\") "
Nov 03 21:22:18.423924 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:18.423756    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aed4032-16a6-4a43-b278-50e496299a92-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aed4032-16a6-4a43-b278-50e496299a92-maas-api-tls\") pod \"4aed4032-16a6-4a43-b278-50e496299a92\" (UID: \"4aed4032-16a6-4a43-b278-50e496299a92\") "
Nov 03 21:22:18.423924 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:18.423810    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aed4032-16a6-4a43-b278-50e496299a92-kube-api-access-2294b\" (UniqueName: \"kubernetes.io/projected/4aed4032-16a6-4a43-b278-50e496299a92-kube-api-access-2294b\") pod \"4aed4032-16a6-4a43-b278-50e496299a92\" (UID: \"4aed4032-16a6-4a43-b278-50e496299a92\") "
Nov 03 21:22:18.423924 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:18.423856    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aed4032-16a6-4a43-b278-50e496299a92-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aed4032-16a6-4a43-b278-50e496299a92-metrics-certs\") pod \"4aed4032-16a6-4a43-b278-50e496299a92\" (UID: \"4aed4032-16a6-4a43-b278-50e496299a92\") "
Nov 03 21:22:18.424088 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:18.424032    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aed4032-16a6-4a43-b278-50e496299a92-openshift-service-ca" pod "4aed4032-16a6-4a43-b278-50e496299a92" (UID: "4aed4032-16a6-4a43-b278-50e496299a92"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:22:18.424238 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:18.424221    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aed4032-16a6-4a43-b278-50e496299a92-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:18.426178 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:18.426154    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aed4032-16a6-4a43-b278-50e496299a92-maas-api-tls" pod "4aed4032-16a6-4a43-b278-50e496299a92" (UID: "4aed4032-16a6-4a43-b278-50e496299a92"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:22:18.426273 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:18.426201    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aed4032-16a6-4a43-b278-50e496299a92-metrics-certs" pod "4aed4032-16a6-4a43-b278-50e496299a92" (UID: "4aed4032-16a6-4a43-b278-50e496299a92"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:22:18.426365 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:18.426347    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aed4032-16a6-4a43-b278-50e496299a92-kube-api-access-2294b" pod "4aed4032-16a6-4a43-b278-50e496299a92" (UID: "4aed4032-16a6-4a43-b278-50e496299a92"). InnerVolumeSpecName "kube-api-access-2294b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:22:18.524752 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:18.524717    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aed4032-16a6-4a43-b278-50e496299a92-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:18.524752 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:18.524745    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2294b\" (UniqueName: \"kubernetes.io/projected/4aed4032-16a6-4a43-b278-50e496299a92-kube-api-access-2294b\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:18.524752 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:18.524755    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aed4032-16a6-4a43-b278-50e496299a92-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:18.798918 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:18.798883    4421 generic.go:356] "Generic (PLEG): container finished" podID="4aed4032-16a6-4a43-b278-50e496299a92" containerID="729529ec01f6a2b6b2cc9287b29540257cd54820464e4f005a6c80c9fae54cb0" exitCode=0
Nov 03 21:22:18.799436 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:18.798946    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6"
Nov 03 21:22:18.799436 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:18.798976    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6" event={"ID":"4aed4032-16a6-4a43-b278-50e496299a92","Type":"ContainerDied","Data":"729529ec01f6a2b6b2cc9287b29540257cd54820464e4f005a6c80c9fae54cb0"}
Nov 03 21:22:18.799436 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:18.799063    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6" event={"ID":"4aed4032-16a6-4a43-b278-50e496299a92","Type":"ContainerDied","Data":"006340b8b0c66cbd2b139dcb4d06682467e113180743b111d84c2bb113894cdc"}
Nov 03 21:22:18.799436 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:18.799090    4421 scope.go:122] "RemoveContainer" containerID="729529ec01f6a2b6b2cc9287b29540257cd54820464e4f005a6c80c9fae54cb0"
Nov 03 21:22:18.810015 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:18.809991    4421 scope.go:122] "RemoveContainer" containerID="729529ec01f6a2b6b2cc9287b29540257cd54820464e4f005a6c80c9fae54cb0"
Nov 03 21:22:18.810317 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:22:18.810276    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"729529ec01f6a2b6b2cc9287b29540257cd54820464e4f005a6c80c9fae54cb0\": container with ID starting with 729529ec01f6a2b6b2cc9287b29540257cd54820464e4f005a6c80c9fae54cb0 not found: ID does not exist" containerID="729529ec01f6a2b6b2cc9287b29540257cd54820464e4f005a6c80c9fae54cb0"
Nov 03 21:22:18.810432 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:18.810315    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"729529ec01f6a2b6b2cc9287b29540257cd54820464e4f005a6c80c9fae54cb0"} err="failed to get container status \"729529ec01f6a2b6b2cc9287b29540257cd54820464e4f005a6c80c9fae54cb0\": rpc error: code = NotFound desc = could not find container \"729529ec01f6a2b6b2cc9287b29540257cd54820464e4f005a6c80c9fae54cb0\": container with ID starting with 729529ec01f6a2b6b2cc9287b29540257cd54820464e4f005a6c80c9fae54cb0 not found: ID does not exist"
Nov 03 21:22:18.824578 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:18.824551    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6"]
Nov 03 21:22:18.831664 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:18.831613    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-d9b592-64f876bc-9b7q6"]
Nov 03 21:22:18.848833 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:18.848801    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"]
Nov 03 21:22:18.849072 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:18.849048    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv" podUID="d668277e-62c6-4f0e-ace6-a6b623f039f2" containerName="istio-proxy" containerID="cri-o://d27de194b887b1cad8be2525995604935822293f13566e98659f1f987a4b3a76" gracePeriod=30
Nov 03 21:22:19.628224 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:22:19.628188    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:22:20.013775 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:20.013742    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aed4032-16a6-4a43-b278-50e496299a92" path="/var/lib/kubelet/pods/4aed4032-16a6-4a43-b278-50e496299a92/volumes"
Nov 03 21:22:22.635775 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:22.635736    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v" podUID="a03cb02c-8dcc-4a83-aba3-6c1309c86212" containerName="payload-processing" containerID="cri-o://66684d68ac885f794b9d3e704977ed41d6cc3b1a18143135fa11194b3a1a2fb7" gracePeriod=25
Nov 03 21:22:22.653411 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:22.653381    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg" podUID="24eef40b-2c73-43c7-8357-1f03e193f5f6" containerName="payload-pre-processing" containerID="cri-o://ad0aec55054bd99c0e78da0ce51408e8af137ace6eb582df7dc4bafdc8f12649" gracePeriod=25
Nov 03 21:22:22.825517 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:22.825472    4421 generic.go:356] "Generic (PLEG): container finished" podID="24eef40b-2c73-43c7-8357-1f03e193f5f6" containerID="ad0aec55054bd99c0e78da0ce51408e8af137ace6eb582df7dc4bafdc8f12649" exitCode=0
Nov 03 21:22:22.825697 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:22.825516    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg" event={"ID":"24eef40b-2c73-43c7-8357-1f03e193f5f6","Type":"ContainerDied","Data":"ad0aec55054bd99c0e78da0ce51408e8af137ace6eb582df7dc4bafdc8f12649"}
Nov 03 21:22:22.827090 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:22.827068    4421 generic.go:356] "Generic (PLEG): container finished" podID="a03cb02c-8dcc-4a83-aba3-6c1309c86212" containerID="66684d68ac885f794b9d3e704977ed41d6cc3b1a18143135fa11194b3a1a2fb7" exitCode=0
Nov 03 21:22:22.827229 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:22.827136    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v" event={"ID":"a03cb02c-8dcc-4a83-aba3-6c1309c86212","Type":"ContainerDied","Data":"66684d68ac885f794b9d3e704977ed41d6cc3b1a18143135fa11194b3a1a2fb7"}
Nov 03 21:22:22.898339 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:22.898277    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v"
Nov 03 21:22:22.931039 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:22.931016    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg"
Nov 03 21:22:22.969114 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:22.969084    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a03cb02c-8dcc-4a83-aba3-6c1309c86212-kube-api-access-qj44b\" (UniqueName: \"kubernetes.io/projected/a03cb02c-8dcc-4a83-aba3-6c1309c86212-kube-api-access-qj44b\") pod \"a03cb02c-8dcc-4a83-aba3-6c1309c86212\" (UID: \"a03cb02c-8dcc-4a83-aba3-6c1309c86212\") "
Nov 03 21:22:22.969266 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:22.969132    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a03cb02c-8dcc-4a83-aba3-6c1309c86212-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a03cb02c-8dcc-4a83-aba3-6c1309c86212-plugins-config-volume\") pod \"a03cb02c-8dcc-4a83-aba3-6c1309c86212\" (UID: \"a03cb02c-8dcc-4a83-aba3-6c1309c86212\") "
Nov 03 21:22:22.969266 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:22.969197    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24eef40b-2c73-43c7-8357-1f03e193f5f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24eef40b-2c73-43c7-8357-1f03e193f5f6-plugins-config-volume\") pod \"24eef40b-2c73-43c7-8357-1f03e193f5f6\" (UID: \"24eef40b-2c73-43c7-8357-1f03e193f5f6\") "
Nov 03 21:22:22.969354 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:22.969266    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24eef40b-2c73-43c7-8357-1f03e193f5f6-kube-api-access-dxfk2\" (UniqueName: \"kubernetes.io/projected/24eef40b-2c73-43c7-8357-1f03e193f5f6-kube-api-access-dxfk2\") pod \"24eef40b-2c73-43c7-8357-1f03e193f5f6\" (UID: \"24eef40b-2c73-43c7-8357-1f03e193f5f6\") "
Nov 03 21:22:22.969650 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:22.969625    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a03cb02c-8dcc-4a83-aba3-6c1309c86212-plugins-config-volume" pod "a03cb02c-8dcc-4a83-aba3-6c1309c86212" (UID: "a03cb02c-8dcc-4a83-aba3-6c1309c86212"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:22:22.969781 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:22.969698    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24eef40b-2c73-43c7-8357-1f03e193f5f6-plugins-config-volume" pod "24eef40b-2c73-43c7-8357-1f03e193f5f6" (UID: "24eef40b-2c73-43c7-8357-1f03e193f5f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:22:22.971517 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:22.971488    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03cb02c-8dcc-4a83-aba3-6c1309c86212-kube-api-access-qj44b" pod "a03cb02c-8dcc-4a83-aba3-6c1309c86212" (UID: "a03cb02c-8dcc-4a83-aba3-6c1309c86212"). InnerVolumeSpecName "kube-api-access-qj44b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:22:22.971649 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:22.971520    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24eef40b-2c73-43c7-8357-1f03e193f5f6-kube-api-access-dxfk2" pod "24eef40b-2c73-43c7-8357-1f03e193f5f6" (UID: "24eef40b-2c73-43c7-8357-1f03e193f5f6"). InnerVolumeSpecName "kube-api-access-dxfk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:22:23.070891 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:23.070809    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24eef40b-2c73-43c7-8357-1f03e193f5f6-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:23.070891 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:23.070848    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxfk2\" (UniqueName: \"kubernetes.io/projected/24eef40b-2c73-43c7-8357-1f03e193f5f6-kube-api-access-dxfk2\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:23.070891 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:23.070867    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj44b\" (UniqueName: \"kubernetes.io/projected/a03cb02c-8dcc-4a83-aba3-6c1309c86212-kube-api-access-qj44b\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:23.070891 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:23.070885    4421 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a03cb02c-8dcc-4a83-aba3-6c1309c86212-plugins-config-volume\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:23.834205 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:23.834170    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg"
Nov 03 21:22:23.834646 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:23.834200    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg" event={"ID":"24eef40b-2c73-43c7-8357-1f03e193f5f6","Type":"ContainerDied","Data":"fc027dc434169df1a407e383f4e4fc8537f8ac7432a138b283dda57f2b9940ad"}
Nov 03 21:22:23.834646 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:23.834263    4421 scope.go:122] "RemoveContainer" containerID="ad0aec55054bd99c0e78da0ce51408e8af137ace6eb582df7dc4bafdc8f12649"
Nov 03 21:22:23.835829 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:23.835802    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v" event={"ID":"a03cb02c-8dcc-4a83-aba3-6c1309c86212","Type":"ContainerDied","Data":"2332d7d1d6e191257ace9b24d37914b5ca89d09bf2a1b10b6134f93a9ffa8e5f"}
Nov 03 21:22:23.835939 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:23.835810    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v"
Nov 03 21:22:23.844863 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:23.844847    4421 scope.go:122] "RemoveContainer" containerID="66684d68ac885f794b9d3e704977ed41d6cc3b1a18143135fa11194b3a1a2fb7"
Nov 03 21:22:23.856269 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:23.856248    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg"]
Nov 03 21:22:23.862201 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:23.862180    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-d9b592-68fc9tkwmg"]
Nov 03 21:22:23.873199 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:23.873177    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v"]
Nov 03 21:22:23.877819 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:23.877796    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-d9b592-5f5dfc877ccj5v"]
Nov 03 21:22:24.014950 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.014921    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24eef40b-2c73-43c7-8357-1f03e193f5f6" path="/var/lib/kubelet/pods/24eef40b-2c73-43c7-8357-1f03e193f5f6/volumes"
Nov 03 21:22:24.015304 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.015291    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a03cb02c-8dcc-4a83-aba3-6c1309c86212" path="/var/lib/kubelet/pods/a03cb02c-8dcc-4a83-aba3-6c1309c86212/volumes"
Nov 03 21:22:24.108779 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.108753    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:22:24.181948 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.181922    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d668277e-62c6-4f0e-ace6-a6b623f039f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d668277e-62c6-4f0e-ace6-a6b623f039f2-istiod-ca-cert\") pod \"d668277e-62c6-4f0e-ace6-a6b623f039f2\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") "
Nov 03 21:22:24.182134 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.181954    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-envoy\") pod \"d668277e-62c6-4f0e-ace6-a6b623f039f2\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") "
Nov 03 21:22:24.182134 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.181981    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-token\" (UniqueName: \"kubernetes.io/projected/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-token\") pod \"d668277e-62c6-4f0e-ace6-a6b623f039f2\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") "
Nov 03 21:22:24.182134 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.182019    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-workload-certs\") pod \"d668277e-62c6-4f0e-ace6-a6b623f039f2\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") "
Nov 03 21:22:24.182293 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.182144    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-credential-socket\") pod \"d668277e-62c6-4f0e-ace6-a6b623f039f2\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") "
Nov 03 21:22:24.182293 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.182165    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-podinfo\") pod \"d668277e-62c6-4f0e-ace6-a6b623f039f2\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") "
Nov 03 21:22:24.182293 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.182190    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d668277e-62c6-4f0e-ace6-a6b623f039f2-kube-api-access-x6xbr\" (UniqueName: \"kubernetes.io/projected/d668277e-62c6-4f0e-ace6-a6b623f039f2-kube-api-access-x6xbr\") pod \"d668277e-62c6-4f0e-ace6-a6b623f039f2\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") "
Nov 03 21:22:24.182293 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.182251    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-workload-socket\") pod \"d668277e-62c6-4f0e-ace6-a6b623f039f2\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") "
Nov 03 21:22:24.182293 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.182291    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-data\") pod \"d668277e-62c6-4f0e-ace6-a6b623f039f2\" (UID: \"d668277e-62c6-4f0e-ace6-a6b623f039f2\") "
Nov 03 21:22:24.182568 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.182373    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-workload-certs" pod "d668277e-62c6-4f0e-ace6-a6b623f039f2" (UID: "d668277e-62c6-4f0e-ace6-a6b623f039f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:24.182568 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.182455    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-credential-socket" pod "d668277e-62c6-4f0e-ace6-a6b623f039f2" (UID: "d668277e-62c6-4f0e-ace6-a6b623f039f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:24.182568 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.182452    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d668277e-62c6-4f0e-ace6-a6b623f039f2-istiod-ca-cert" pod "d668277e-62c6-4f0e-ace6-a6b623f039f2" (UID: "d668277e-62c6-4f0e-ace6-a6b623f039f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:22:24.182785 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.182643    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-workload-socket" pod "d668277e-62c6-4f0e-ace6-a6b623f039f2" (UID: "d668277e-62c6-4f0e-ace6-a6b623f039f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:24.182785 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.182662    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-data" pod "d668277e-62c6-4f0e-ace6-a6b623f039f2" (UID: "d668277e-62c6-4f0e-ace6-a6b623f039f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:24.182889 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.182711    4421 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-credential-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:24.182889 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.182885    4421 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d668277e-62c6-4f0e-ace6-a6b623f039f2-istiod-ca-cert\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:24.182971 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.182900    4421 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-workload-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:24.184356 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.184329    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-envoy" pod "d668277e-62c6-4f0e-ace6-a6b623f039f2" (UID: "d668277e-62c6-4f0e-ace6-a6b623f039f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 21:22:24.184495 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.184422    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-token" pod "d668277e-62c6-4f0e-ace6-a6b623f039f2" (UID: "d668277e-62c6-4f0e-ace6-a6b623f039f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:22:24.185045 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.185013    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-podinfo" pod "d668277e-62c6-4f0e-ace6-a6b623f039f2" (UID: "d668277e-62c6-4f0e-ace6-a6b623f039f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 21:22:24.185156 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.185052    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d668277e-62c6-4f0e-ace6-a6b623f039f2-kube-api-access-x6xbr" pod "d668277e-62c6-4f0e-ace6-a6b623f039f2" (UID: "d668277e-62c6-4f0e-ace6-a6b623f039f2"). InnerVolumeSpecName "kube-api-access-x6xbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:22:24.283877 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.283799    4421 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-token\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:24.283877 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.283827    4421 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-podinfo\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:24.283877 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.283837    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6xbr\" (UniqueName: \"kubernetes.io/projected/d668277e-62c6-4f0e-ace6-a6b623f039f2-kube-api-access-x6xbr\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:24.283877 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.283847    4421 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-workload-socket\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:24.283877 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.283855    4421 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-data\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:24.283877 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.283864    4421 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d668277e-62c6-4f0e-ace6-a6b623f039f2-istio-envoy\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:22:24.842792 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.842759    4421 generic.go:356] "Generic (PLEG): container finished" podID="d668277e-62c6-4f0e-ace6-a6b623f039f2" containerID="d27de194b887b1cad8be2525995604935822293f13566e98659f1f987a4b3a76" exitCode=0
Nov 03 21:22:24.843323 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.842827    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"
Nov 03 21:22:24.843323 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.842836    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv" event={"ID":"d668277e-62c6-4f0e-ace6-a6b623f039f2","Type":"ContainerDied","Data":"d27de194b887b1cad8be2525995604935822293f13566e98659f1f987a4b3a76"}
Nov 03 21:22:24.843323 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.842926    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv" event={"ID":"d668277e-62c6-4f0e-ace6-a6b623f039f2","Type":"ContainerDied","Data":"e037b2c82bada0f0af6e1cdcb71cbcadd6a8b47b821b59532d86ab11f3571ece"}
Nov 03 21:22:24.843323 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.842963    4421 scope.go:122] "RemoveContainer" containerID="d27de194b887b1cad8be2525995604935822293f13566e98659f1f987a4b3a76"
Nov 03 21:22:24.852278 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.852257    4421 scope.go:122] "RemoveContainer" containerID="1d191b6e395a6c4fff1fd4355c23c21721cbe1d6809cb0cfbfd495264f5bc2b9"
Nov 03 21:22:24.861070 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.861051    4421 scope.go:122] "RemoveContainer" containerID="d27de194b887b1cad8be2525995604935822293f13566e98659f1f987a4b3a76"
Nov 03 21:22:24.861320 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:22:24.861304    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d27de194b887b1cad8be2525995604935822293f13566e98659f1f987a4b3a76\": container with ID starting with d27de194b887b1cad8be2525995604935822293f13566e98659f1f987a4b3a76 not found: ID does not exist" containerID="d27de194b887b1cad8be2525995604935822293f13566e98659f1f987a4b3a76"
Nov 03 21:22:24.861396 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.861328    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d27de194b887b1cad8be2525995604935822293f13566e98659f1f987a4b3a76"} err="failed to get container status \"d27de194b887b1cad8be2525995604935822293f13566e98659f1f987a4b3a76\": rpc error: code = NotFound desc = could not find container \"d27de194b887b1cad8be2525995604935822293f13566e98659f1f987a4b3a76\": container with ID starting with d27de194b887b1cad8be2525995604935822293f13566e98659f1f987a4b3a76 not found: ID does not exist"
Nov 03 21:22:24.861396 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.861342    4421 scope.go:122] "RemoveContainer" containerID="1d191b6e395a6c4fff1fd4355c23c21721cbe1d6809cb0cfbfd495264f5bc2b9"
Nov 03 21:22:24.861539 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:22:24.861523    4421 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d191b6e395a6c4fff1fd4355c23c21721cbe1d6809cb0cfbfd495264f5bc2b9\": container with ID starting with 1d191b6e395a6c4fff1fd4355c23c21721cbe1d6809cb0cfbfd495264f5bc2b9 not found: ID does not exist" containerID="1d191b6e395a6c4fff1fd4355c23c21721cbe1d6809cb0cfbfd495264f5bc2b9"
Nov 03 21:22:24.861576 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.861543    4421 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d191b6e395a6c4fff1fd4355c23c21721cbe1d6809cb0cfbfd495264f5bc2b9"} err="failed to get container status \"1d191b6e395a6c4fff1fd4355c23c21721cbe1d6809cb0cfbfd495264f5bc2b9\": rpc error: code = NotFound desc = could not find container \"1d191b6e395a6c4fff1fd4355c23c21721cbe1d6809cb0cfbfd495264f5bc2b9\": container with ID starting with 1d191b6e395a6c4fff1fd4355c23c21721cbe1d6809cb0cfbfd495264f5bc2b9 not found: ID does not exist"
Nov 03 21:22:24.865420 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.865397    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"]
Nov 03 21:22:24.871680 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:24.871660    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-d9b592-openshift-default-c7d56c589-zz2lv"]
Nov 03 21:22:26.014251 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:22:26.014216    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d668277e-62c6-4f0e-ace6-a6b623f039f2" path="/var/lib/kubelet/pods/d668277e-62c6-4f0e-ace6-a6b623f039f2/volumes"
Nov 03 21:22:49.628826 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:22:49.628744    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:23:19.627991 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:23:19.627953    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:23:36.808853 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:36.808817    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9p95v"]
Nov 03 21:23:36.809366 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:36.809206    4421 cpu_manager.go:414] "containerMap: removing container" podUID="4aed4032-16a6-4a43-b278-50e496299a92" containerName="maas-api"
Nov 03 21:23:36.809366 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:36.809216    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aed4032-16a6-4a43-b278-50e496299a92" containerName="maas-api"
Nov 03 21:23:36.809366 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:36.809230    4421 cpu_manager.go:414] "containerMap: removing container" podUID="24eef40b-2c73-43c7-8357-1f03e193f5f6" containerName="payload-pre-processing"
Nov 03 21:23:36.809366 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:36.809235    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24eef40b-2c73-43c7-8357-1f03e193f5f6" containerName="payload-pre-processing"
Nov 03 21:23:36.809366 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:36.809245    4421 cpu_manager.go:414] "containerMap: removing container" podUID="d668277e-62c6-4f0e-ace6-a6b623f039f2" containerName="istio-proxy"
Nov 03 21:23:36.809366 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:36.809249    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d668277e-62c6-4f0e-ace6-a6b623f039f2" containerName="istio-proxy"
Nov 03 21:23:36.809366 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:36.809265    4421 cpu_manager.go:414] "containerMap: removing container" podUID="a03cb02c-8dcc-4a83-aba3-6c1309c86212" containerName="payload-processing"
Nov 03 21:23:36.809366 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:36.809270    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a03cb02c-8dcc-4a83-aba3-6c1309c86212" containerName="payload-processing"
Nov 03 21:23:36.809366 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:36.809276    4421 cpu_manager.go:414] "containerMap: removing container" podUID="d668277e-62c6-4f0e-ace6-a6b623f039f2" containerName="istio-proxy"
Nov 03 21:23:36.809366 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:36.809280    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d668277e-62c6-4f0e-ace6-a6b623f039f2" containerName="istio-proxy"
Nov 03 21:23:36.809366 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:36.809347    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aed4032-16a6-4a43-b278-50e496299a92" containerName="maas-api"
Nov 03 21:23:36.809366 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:36.809355    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="24eef40b-2c73-43c7-8357-1f03e193f5f6" containerName="payload-pre-processing"
Nov 03 21:23:36.809366 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:36.809362    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d668277e-62c6-4f0e-ace6-a6b623f039f2" containerName="istio-proxy"
Nov 03 21:23:36.809366 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:36.809370    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="a03cb02c-8dcc-4a83-aba3-6c1309c86212" containerName="payload-processing"
Nov 03 21:23:36.812486 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:36.812470    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9p95v"
Nov 03 21:23:36.816528 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:36.816503    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pl6rl\""
Nov 03 21:23:36.827745 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:36.827722    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9p95v"]
Nov 03 21:23:36.887334 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:36.887297    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tbrd\" (UniqueName: \"kubernetes.io/projected/5c786253-34d0-4b8b-8924-792d8cbc3d9f-kube-api-access-9tbrd\") pod \"kuadrant-operator-controller-manager-678d87766f-9p95v\" (UID: \"5c786253-34d0-4b8b-8924-792d8cbc3d9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9p95v"
Nov 03 21:23:36.887498 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:36.887419    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c786253-34d0-4b8b-8924-792d8cbc3d9f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-9p95v\" (UID: \"5c786253-34d0-4b8b-8924-792d8cbc3d9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9p95v"
Nov 03 21:23:36.988576 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:36.988543    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tbrd\" (UniqueName: \"kubernetes.io/projected/5c786253-34d0-4b8b-8924-792d8cbc3d9f-kube-api-access-9tbrd\") pod \"kuadrant-operator-controller-manager-678d87766f-9p95v\" (UID: \"5c786253-34d0-4b8b-8924-792d8cbc3d9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9p95v"
Nov 03 21:23:36.988757 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:36.988627    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c786253-34d0-4b8b-8924-792d8cbc3d9f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-9p95v\" (UID: \"5c786253-34d0-4b8b-8924-792d8cbc3d9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9p95v"
Nov 03 21:23:36.988950 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:36.988933    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c786253-34d0-4b8b-8924-792d8cbc3d9f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-9p95v\" (UID: \"5c786253-34d0-4b8b-8924-792d8cbc3d9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9p95v"
Nov 03 21:23:37.001293 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:37.001264    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tbrd\" (UniqueName: \"kubernetes.io/projected/5c786253-34d0-4b8b-8924-792d8cbc3d9f-kube-api-access-9tbrd\") pod \"kuadrant-operator-controller-manager-678d87766f-9p95v\" (UID: \"5c786253-34d0-4b8b-8924-792d8cbc3d9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9p95v"
Nov 03 21:23:37.125522 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:37.125424    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9p95v"
Nov 03 21:23:37.407103 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:37.407076    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9p95v"]
Nov 03 21:23:37.409379 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:23:37.409350    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c786253_34d0_4b8b_8924_792d8cbc3d9f.slice/crio-55742972f12085d77781cbf854152c739b55aaf3a198918c3f891432063d0a45 WatchSource:0}: Error finding container 55742972f12085d77781cbf854152c739b55aaf3a198918c3f891432063d0a45: Status 404 returned error can't find the container with id 55742972f12085d77781cbf854152c739b55aaf3a198918c3f891432063d0a45
Nov 03 21:23:38.173276 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:38.173242    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9p95v" event={"ID":"5c786253-34d0-4b8b-8924-792d8cbc3d9f","Type":"ContainerStarted","Data":"55742972f12085d77781cbf854152c739b55aaf3a198918c3f891432063d0a45"}
Nov 03 21:23:46.217677 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:46.217638    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9p95v" event={"ID":"5c786253-34d0-4b8b-8924-792d8cbc3d9f","Type":"ContainerStarted","Data":"a077809dcdc3fd8647c7560e9851112de2de9bfe5680cfee26f0d94659edb91c"}
Nov 03 21:23:46.218149 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:46.217769    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9p95v"
Nov 03 21:23:46.243985 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:46.243931    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9p95v" podStartSLOduration=2.443460197 podStartE2EDuration="10.243916742s" totalImagesPullingTime="7.800456545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:23:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:23:46.242469478 +0000 UTC m=+4956.375597941" watchObservedRunningTime="2026-11-03 21:23:46.243916742 +0000 UTC m=+4956.377045203"
Nov 03 21:23:49.628080 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:23:49.628043    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:23:57.226750 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:23:57.226716    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-9p95v"
Nov 03 21:24:19.628783 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:24:19.628691    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:24:37.235966 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:37.235910    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b4d8b8448-9j7fq"]
Nov 03 21:24:37.237133 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:37.237108    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="d668277e-62c6-4f0e-ace6-a6b623f039f2" containerName="istio-proxy"
Nov 03 21:24:37.240460 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:37.240435    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-9j7fq"
Nov 03 21:24:37.252511 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:37.251112    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b4d8b8448-9j7fq"]
Nov 03 21:24:37.267434 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:37.267398    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09d9609c-6fa8-4eaf-b281-d3c55fe1950a-metrics-certs\") pod \"maas-api-6b4d8b8448-9j7fq\" (UID: \"09d9609c-6fa8-4eaf-b281-d3c55fe1950a\") " pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-9j7fq"
Nov 03 21:24:37.267634 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:37.267494    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09d9609c-6fa8-4eaf-b281-d3c55fe1950a-maas-api-tls\") pod \"maas-api-6b4d8b8448-9j7fq\" (UID: \"09d9609c-6fa8-4eaf-b281-d3c55fe1950a\") " pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-9j7fq"
Nov 03 21:24:37.267735 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:37.267671    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg7qh\" (UniqueName: \"kubernetes.io/projected/09d9609c-6fa8-4eaf-b281-d3c55fe1950a-kube-api-access-hg7qh\") pod \"maas-api-6b4d8b8448-9j7fq\" (UID: \"09d9609c-6fa8-4eaf-b281-d3c55fe1950a\") " pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-9j7fq"
Nov 03 21:24:37.267802 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:37.267785    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09d9609c-6fa8-4eaf-b281-d3c55fe1950a-openshift-service-ca\") pod \"maas-api-6b4d8b8448-9j7fq\" (UID: \"09d9609c-6fa8-4eaf-b281-d3c55fe1950a\") " pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-9j7fq"
Nov 03 21:24:37.369200 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:37.369165    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09d9609c-6fa8-4eaf-b281-d3c55fe1950a-maas-api-tls\") pod \"maas-api-6b4d8b8448-9j7fq\" (UID: \"09d9609c-6fa8-4eaf-b281-d3c55fe1950a\") " pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-9j7fq"
Nov 03 21:24:37.369398 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:37.369208    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg7qh\" (UniqueName: \"kubernetes.io/projected/09d9609c-6fa8-4eaf-b281-d3c55fe1950a-kube-api-access-hg7qh\") pod \"maas-api-6b4d8b8448-9j7fq\" (UID: \"09d9609c-6fa8-4eaf-b281-d3c55fe1950a\") " pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-9j7fq"
Nov 03 21:24:37.369398 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:37.369244    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09d9609c-6fa8-4eaf-b281-d3c55fe1950a-openshift-service-ca\") pod \"maas-api-6b4d8b8448-9j7fq\" (UID: \"09d9609c-6fa8-4eaf-b281-d3c55fe1950a\") " pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-9j7fq"
Nov 03 21:24:37.369398 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:37.369291    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09d9609c-6fa8-4eaf-b281-d3c55fe1950a-metrics-certs\") pod \"maas-api-6b4d8b8448-9j7fq\" (UID: \"09d9609c-6fa8-4eaf-b281-d3c55fe1950a\") " pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-9j7fq"
Nov 03 21:24:37.370076 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:37.369979    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09d9609c-6fa8-4eaf-b281-d3c55fe1950a-openshift-service-ca\") pod \"maas-api-6b4d8b8448-9j7fq\" (UID: \"09d9609c-6fa8-4eaf-b281-d3c55fe1950a\") " pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-9j7fq"
Nov 03 21:24:37.371845 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:37.371825    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09d9609c-6fa8-4eaf-b281-d3c55fe1950a-metrics-certs\") pod \"maas-api-6b4d8b8448-9j7fq\" (UID: \"09d9609c-6fa8-4eaf-b281-d3c55fe1950a\") " pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-9j7fq"
Nov 03 21:24:37.371994 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:37.371976    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09d9609c-6fa8-4eaf-b281-d3c55fe1950a-maas-api-tls\") pod \"maas-api-6b4d8b8448-9j7fq\" (UID: \"09d9609c-6fa8-4eaf-b281-d3c55fe1950a\") " pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-9j7fq"
Nov 03 21:24:37.381676 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:37.381653    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg7qh\" (UniqueName: \"kubernetes.io/projected/09d9609c-6fa8-4eaf-b281-d3c55fe1950a-kube-api-access-hg7qh\") pod \"maas-api-6b4d8b8448-9j7fq\" (UID: \"09d9609c-6fa8-4eaf-b281-d3c55fe1950a\") " pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-9j7fq"
Nov 03 21:24:37.561183 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:37.561103    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-9j7fq"
Nov 03 21:24:37.841915 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:37.841890    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b4d8b8448-9j7fq"]
Nov 03 21:24:37.844534 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:24:37.844484    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d9609c_6fa8_4eaf_b281_d3c55fe1950a.slice/crio-e63690bc6d7dfe1108b06cae734ffd618da10a4a972cfe5fee24ea385d0fb987 WatchSource:0}: Error finding container e63690bc6d7dfe1108b06cae734ffd618da10a4a972cfe5fee24ea385d0fb987: Status 404 returned error can't find the container with id e63690bc6d7dfe1108b06cae734ffd618da10a4a972cfe5fee24ea385d0fb987
Nov 03 21:24:38.563128 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:38.563092    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-9j7fq" event={"ID":"09d9609c-6fa8-4eaf-b281-d3c55fe1950a","Type":"ContainerStarted","Data":"18a2cfd3ca320ab255a8739bb0fc9fcdc4e413ca46899ee01990db400f9a0ceb"}
Nov 03 21:24:38.563562 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:38.563155    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-9j7fq" event={"ID":"09d9609c-6fa8-4eaf-b281-d3c55fe1950a","Type":"ContainerStarted","Data":"e63690bc6d7dfe1108b06cae734ffd618da10a4a972cfe5fee24ea385d0fb987"}
Nov 03 21:24:38.563562 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:38.563215    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-9j7fq"
Nov 03 21:24:38.583933 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:38.583873    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-9j7fq" podStartSLOduration=1.057839912 podStartE2EDuration="1.583858036s" totalImagesPullingTime="526.018124ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:24:38.582201114 +0000 UTC m=+5008.715329579" watchObservedRunningTime="2026-11-03 21:24:38.583858036 +0000 UTC m=+5008.716986498"
Nov 03 21:24:44.574370 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:44.574343    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-9j7fq"
Nov 03 21:24:49.712774 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:24:49.712733    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:24:52.095085 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:52.095049    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6474999cb4-52z85"]
Nov 03 21:24:52.126273 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:52.109170    4421 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6474999cb4-52z85" podUID="e0e46ea6-aded-4950-9550-3c35af396845" containerName="maas-api" containerID="cri-o://15d2013e339e3a3bdebbb8424a0cc090856009e07be63dc2bc94e6aa623e85f7" gracePeriod=30
Nov 03 21:24:53.722514 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:53.722406    4421 generic.go:356] "Generic (PLEG): container finished" podID="e0e46ea6-aded-4950-9550-3c35af396845" containerID="15d2013e339e3a3bdebbb8424a0cc090856009e07be63dc2bc94e6aa623e85f7" exitCode=0
Nov 03 21:24:53.723153 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:53.723127    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6474999cb4-52z85" event={"ID":"e0e46ea6-aded-4950-9550-3c35af396845","Type":"ContainerDied","Data":"15d2013e339e3a3bdebbb8424a0cc090856009e07be63dc2bc94e6aa623e85f7"}
Nov 03 21:24:53.803444 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:53.803408    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6474999cb4-52z85"
Nov 03 21:24:53.887326 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:53.887232    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0e46ea6-aded-4950-9550-3c35af396845-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e46ea6-aded-4950-9550-3c35af396845-maas-api-tls\") pod \"e0e46ea6-aded-4950-9550-3c35af396845\" (UID: \"e0e46ea6-aded-4950-9550-3c35af396845\") "
Nov 03 21:24:53.887502 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:53.887325    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0e46ea6-aded-4950-9550-3c35af396845-kube-api-access-6grdx\" (UniqueName: \"kubernetes.io/projected/e0e46ea6-aded-4950-9550-3c35af396845-kube-api-access-6grdx\") pod \"e0e46ea6-aded-4950-9550-3c35af396845\" (UID: \"e0e46ea6-aded-4950-9550-3c35af396845\") "
Nov 03 21:24:53.887502 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:53.887396    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0e46ea6-aded-4950-9550-3c35af396845-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e46ea6-aded-4950-9550-3c35af396845-openshift-service-ca\") pod \"e0e46ea6-aded-4950-9550-3c35af396845\" (UID: \"e0e46ea6-aded-4950-9550-3c35af396845\") "
Nov 03 21:24:53.887502 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:53.887428    4421 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0e46ea6-aded-4950-9550-3c35af396845-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0e46ea6-aded-4950-9550-3c35af396845-metrics-certs\") pod \"e0e46ea6-aded-4950-9550-3c35af396845\" (UID: \"e0e46ea6-aded-4950-9550-3c35af396845\") "
Nov 03 21:24:53.888148 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:53.888072    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0e46ea6-aded-4950-9550-3c35af396845-openshift-service-ca" pod "e0e46ea6-aded-4950-9550-3c35af396845" (UID: "e0e46ea6-aded-4950-9550-3c35af396845"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:24:53.888462 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:53.888443    4421 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e46ea6-aded-4950-9550-3c35af396845-openshift-service-ca\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:24:53.890679 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:53.890520    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e46ea6-aded-4950-9550-3c35af396845-kube-api-access-6grdx" pod "e0e46ea6-aded-4950-9550-3c35af396845" (UID: "e0e46ea6-aded-4950-9550-3c35af396845"). InnerVolumeSpecName "kube-api-access-6grdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:24:53.890679 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:53.890644    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0e46ea6-aded-4950-9550-3c35af396845-metrics-certs" pod "e0e46ea6-aded-4950-9550-3c35af396845" (UID: "e0e46ea6-aded-4950-9550-3c35af396845"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:24:53.892780 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:53.892742    4421 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0e46ea6-aded-4950-9550-3c35af396845-maas-api-tls" pod "e0e46ea6-aded-4950-9550-3c35af396845" (UID: "e0e46ea6-aded-4950-9550-3c35af396845"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 21:24:53.989426 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:53.989394    4421 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e46ea6-aded-4950-9550-3c35af396845-maas-api-tls\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:24:53.989426 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:53.989425    4421 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6grdx\" (UniqueName: \"kubernetes.io/projected/e0e46ea6-aded-4950-9550-3c35af396845-kube-api-access-6grdx\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:24:53.989654 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:53.989436    4421 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0e46ea6-aded-4950-9550-3c35af396845-metrics-certs\") on node \"ip-10-0-143-48.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:24:54.728829 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:54.728794    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6474999cb4-52z85" event={"ID":"e0e46ea6-aded-4950-9550-3c35af396845","Type":"ContainerDied","Data":"730a11993a8bf9efd89b58309a177f219ffe3190619affb952529b69383d986d"}
Nov 03 21:24:54.728829 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:54.728826    4421 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6474999cb4-52z85"
Nov 03 21:24:54.729367 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:54.728855    4421 scope.go:122] "RemoveContainer" containerID="15d2013e339e3a3bdebbb8424a0cc090856009e07be63dc2bc94e6aa623e85f7"
Nov 03 21:24:54.746913 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:54.746890    4421 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6474999cb4-52z85"]
Nov 03 21:24:54.751461 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:54.751439    4421 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6474999cb4-52z85"]
Nov 03 21:24:56.014215 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:24:56.014171    4421 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0e46ea6-aded-4950-9550-3c35af396845" path="/var/lib/kubelet/pods/e0e46ea6-aded-4950-9550-3c35af396845/volumes"
Nov 03 21:25:19.643846 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:25:19.643797    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:25:49.643097 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:25:49.642993    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:26:19.640897 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:26:19.640849    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:26:31.029735 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:26:31.029693    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xfwgr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:26:31.030481 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:26:31.030445    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr" podUID="bc25e3fb-91ba-4d75-92d3-d1aab2feb033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:26:49.784853 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:26:49.784809    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:27:12.494625 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:27:12.493724    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-6bhmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:27:12.494625 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:27:12.493824    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6bhmr" podUID="9d6e2ee7-7d48-4879-b6fa-f303b5d24697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:27:16.037509 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:27:16.037464    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xfwgr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:27:16.038352 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:27:16.038318    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xfwgr" podUID="bc25e3fb-91ba-4d75-92d3-d1aab2feb033" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:27:19.803069 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:27:19.802830    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:27:49.628283 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:27:49.628246    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:28:19.627658 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:28:19.627592    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:28:27.929625 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:28:27.928732    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-cd5ll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:28:27.929625 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:28:27.928874    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cd5ll" podUID="b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:28:28.604624 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:28:28.601704    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bnt6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:28:28.604624 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:28:28.601780    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k" podUID="9375775d-7b1d-48d0-ab7b-ce8cf03729e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:28:49.629663 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:28:49.629548    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:28:58.608242 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:28:58.607722    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bnt6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:28:58.608242 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:28:58.607796    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k" podUID="9375775d-7b1d-48d0-ab7b-ce8cf03729e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:29:19.627446 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:29:19.627413    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:29:28.618624 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:29:28.617703    4421 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bnt6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:29:28.618624 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:29:28.617783    4421 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bnt6k" podUID="9375775d-7b1d-48d0-ab7b-ce8cf03729e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:29:36.548092 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:29:36.548061    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cfb485dff-mjxfv_6f1f72d4-3cc6-4ae2-b9ac-a399a5b9a34c/authorino/0.log"
Nov 03 21:29:48.304156 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:29:48.304120    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-2725f_1c4bb3bf-caf2-4932-8b65-00189e8d41fb/manager/0.log"
Nov 03 21:29:48.990767 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:29:48.990732    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-687b69bd9-hmtjv_670212d6-aae1-4981-aaf1-5b12df51bc0d/manager/0.log"
Nov 03 21:29:49.216529 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:29:49.216498    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-8lzxb_c23a53b9-ca79-461a-a8f6-20618ffc35aa/server/0.log"
Nov 03 21:29:49.396767 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:29:49.396690    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-99ntv_b347c3ff-a128-4fd2-8c07-dfea5c60a431/manager/0.log"
Nov 03 21:29:49.597837 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:29:49.597794    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-4bbvd_1dcbd7f2-5145-444c-9b8f-7035eb14b64e/proxy/0.log"
Nov 03 21:29:49.627799 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:29:49.627766    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:29:50.675714 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:29:50.675690    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6b4d8b8448-9j7fq_09d9609c-6fa8-4eaf-b281-d3c55fe1950a/maas-api/0.log"
Nov 03 21:29:56.081690 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:29:56.081652    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cfb485dff-mjxfv_6f1f72d4-3cc6-4ae2-b9ac-a399a5b9a34c/authorino/0.log"
Nov 03 21:29:56.683104 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:29:56.683080    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jb45k_7e5c45c5-1f48-4b84-b7ae-244cd6e80b9f/kuadrant-console-plugin/0.log"
Nov 03 21:29:56.913139 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:29:56.913103    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-9p95v_5c786253-34d0-4b8b-8924-792d8cbc3d9f/manager/0.log"
Nov 03 21:29:58.814390 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:29:58.814360    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-594f764c95-z8fjv_b5520324-5b0c-4417-8221-a2cf1c08b881/kube-auth-proxy/0.log"
Nov 03 21:29:58.997243 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:29:58.997210    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-6bhmr_9d6e2ee7-7d48-4879-b6fa-f303b5d24697/istio-proxy/0.log"
Nov 03 21:29:59.382042 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:29:59.382010    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-bnt6k_9375775d-7b1d-48d0-ab7b-ce8cf03729e9/istio-proxy/0.log"
Nov 03 21:29:59.574299 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:29:59.574265    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-cd5ll_b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d/istio-proxy/0.log"
Nov 03 21:30:00.544520 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:00.544488    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-xfwgr_bc25e3fb-91ba-4d75-92d3-d1aab2feb033/istio-proxy/0.log"
Nov 03 21:30:01.091491 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:01.091465    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-5556d898cf-4gn6l_835ed6d1-8b5c-4b49-8316-0fbb1d161e2c/payload-pre-processing/0.log"
Nov 03 21:30:01.269955 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:01.269931    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7dc89df4c4-hhqqf_e78b2b21-5706-465b-8ea1-9f1679e0f4ff/payload-processing/0.log"
Nov 03 21:30:02.426061 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:02.426032    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-bg4kj_40f4d063-2060-49f8-8b17-13526284ed89/main/0.log"
Nov 03 21:30:02.941772 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:02.941746    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-external-token-limit-simulated-kserve-65fc98cd9c-2qdw8_d37ba58a-679f-4489-ab67-001ea89cf66d/main/0.log"
Nov 03 21:30:03.303516 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:03.303441    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qzxvv_cc9f7589-f4e9-4445-95cb-7b0cc446cee4/main/0.log"
Nov 03 21:30:03.665841 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:03.665759    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-4m2zs_7e155365-719f-4ee2-bdd0-08b076f7513c/main/0.log"
Nov 03 21:30:13.750004 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:13.749968    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x6xjk/must-gather-fr5j8"]
Nov 03 21:30:13.750422 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:13.750407    4421 cpu_manager.go:414] "containerMap: removing container" podUID="e0e46ea6-aded-4950-9550-3c35af396845" containerName="maas-api"
Nov 03 21:30:13.750473 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:13.750423    4421 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0e46ea6-aded-4950-9550-3c35af396845" containerName="maas-api"
Nov 03 21:30:13.750511 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:13.750507    4421 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0e46ea6-aded-4950-9550-3c35af396845" containerName="maas-api"
Nov 03 21:30:13.754111 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:13.754095    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x6xjk/must-gather-fr5j8"
Nov 03 21:30:13.763387 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:13.763361    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x6xjk/must-gather-fr5j8"]
Nov 03 21:30:13.765022 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:13.765002    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x6xjk\"/\"openshift-service-ca.crt\""
Nov 03 21:30:13.765141 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:13.765056    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x6xjk\"/\"kube-root-ca.crt\""
Nov 03 21:30:13.765141 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:13.765114    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-x6xjk\"/\"default-dockercfg-zdd4r\""
Nov 03 21:30:13.786444 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:13.786416    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/03f212a0-dc19-4e3b-aa7c-88e97451e317-must-gather-output\") pod \"must-gather-fr5j8\" (UID: \"03f212a0-dc19-4e3b-aa7c-88e97451e317\") " pod="openshift-must-gather-x6xjk/must-gather-fr5j8"
Nov 03 21:30:13.786586 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:13.786473    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq9t6\" (UniqueName: \"kubernetes.io/projected/03f212a0-dc19-4e3b-aa7c-88e97451e317-kube-api-access-cq9t6\") pod \"must-gather-fr5j8\" (UID: \"03f212a0-dc19-4e3b-aa7c-88e97451e317\") " pod="openshift-must-gather-x6xjk/must-gather-fr5j8"
Nov 03 21:30:13.848646 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:13.848618    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fjmcb/must-gather-ltslt"]
Nov 03 21:30:13.852446 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:13.852427    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fjmcb/must-gather-ltslt"
Nov 03 21:30:13.861466 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:13.861444    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fjmcb/must-gather-ltslt"]
Nov 03 21:30:13.864544 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:13.864524    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fjmcb\"/\"kube-root-ca.crt\""
Nov 03 21:30:13.864688 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:13.864574    4421 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fjmcb\"/\"openshift-service-ca.crt\""
Nov 03 21:30:13.864688 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:13.864636    4421 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fjmcb\"/\"default-dockercfg-sl6jl\""
Nov 03 21:30:13.887399 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:13.887371    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5eacb6e5-f8b4-4535-8888-b2049f8a5d1a-must-gather-output\") pod \"must-gather-ltslt\" (UID: \"5eacb6e5-f8b4-4535-8888-b2049f8a5d1a\") " pod="openshift-must-gather-fjmcb/must-gather-ltslt"
Nov 03 21:30:13.887517 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:13.887479    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc7rc\" (UniqueName: \"kubernetes.io/projected/5eacb6e5-f8b4-4535-8888-b2049f8a5d1a-kube-api-access-jc7rc\") pod \"must-gather-ltslt\" (UID: \"5eacb6e5-f8b4-4535-8888-b2049f8a5d1a\") " pod="openshift-must-gather-fjmcb/must-gather-ltslt"
Nov 03 21:30:13.887647 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:13.887531    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/03f212a0-dc19-4e3b-aa7c-88e97451e317-must-gather-output\") pod \"must-gather-fr5j8\" (UID: \"03f212a0-dc19-4e3b-aa7c-88e97451e317\") " pod="openshift-must-gather-x6xjk/must-gather-fr5j8"
Nov 03 21:30:13.887647 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:13.887569    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq9t6\" (UniqueName: \"kubernetes.io/projected/03f212a0-dc19-4e3b-aa7c-88e97451e317-kube-api-access-cq9t6\") pod \"must-gather-fr5j8\" (UID: \"03f212a0-dc19-4e3b-aa7c-88e97451e317\") " pod="openshift-must-gather-x6xjk/must-gather-fr5j8"
Nov 03 21:30:13.887889 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:13.887868    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/03f212a0-dc19-4e3b-aa7c-88e97451e317-must-gather-output\") pod \"must-gather-fr5j8\" (UID: \"03f212a0-dc19-4e3b-aa7c-88e97451e317\") " pod="openshift-must-gather-x6xjk/must-gather-fr5j8"
Nov 03 21:30:13.901779 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:13.901756    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq9t6\" (UniqueName: \"kubernetes.io/projected/03f212a0-dc19-4e3b-aa7c-88e97451e317-kube-api-access-cq9t6\") pod \"must-gather-fr5j8\" (UID: \"03f212a0-dc19-4e3b-aa7c-88e97451e317\") " pod="openshift-must-gather-x6xjk/must-gather-fr5j8"
Nov 03 21:30:13.988973 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:13.988935    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5eacb6e5-f8b4-4535-8888-b2049f8a5d1a-must-gather-output\") pod \"must-gather-ltslt\" (UID: \"5eacb6e5-f8b4-4535-8888-b2049f8a5d1a\") " pod="openshift-must-gather-fjmcb/must-gather-ltslt"
Nov 03 21:30:13.989151 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:13.989012    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc7rc\" (UniqueName: \"kubernetes.io/projected/5eacb6e5-f8b4-4535-8888-b2049f8a5d1a-kube-api-access-jc7rc\") pod \"must-gather-ltslt\" (UID: \"5eacb6e5-f8b4-4535-8888-b2049f8a5d1a\") " pod="openshift-must-gather-fjmcb/must-gather-ltslt"
Nov 03 21:30:13.989315 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:13.989296    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5eacb6e5-f8b4-4535-8888-b2049f8a5d1a-must-gather-output\") pod \"must-gather-ltslt\" (UID: \"5eacb6e5-f8b4-4535-8888-b2049f8a5d1a\") " pod="openshift-must-gather-fjmcb/must-gather-ltslt"
Nov 03 21:30:14.002443 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:14.002384    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc7rc\" (UniqueName: \"kubernetes.io/projected/5eacb6e5-f8b4-4535-8888-b2049f8a5d1a-kube-api-access-jc7rc\") pod \"must-gather-ltslt\" (UID: \"5eacb6e5-f8b4-4535-8888-b2049f8a5d1a\") " pod="openshift-must-gather-fjmcb/must-gather-ltslt"
Nov 03 21:30:14.069767 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:14.069738    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x6xjk/must-gather-fr5j8"
Nov 03 21:30:14.167947 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:14.167920    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fjmcb/must-gather-ltslt"
Nov 03 21:30:14.344301 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:14.344279    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x6xjk/must-gather-fr5j8"]
Nov 03 21:30:14.347815 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:30:14.347787    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03f212a0_dc19_4e3b_aa7c_88e97451e317.slice/crio-14f4b428ffc2f19ee659753c31b906cf7f8d04ab2e3ec7e141a2fd9a0092d03f WatchSource:0}: Error finding container 14f4b428ffc2f19ee659753c31b906cf7f8d04ab2e3ec7e141a2fd9a0092d03f: Status 404 returned error can't find the container with id 14f4b428ffc2f19ee659753c31b906cf7f8d04ab2e3ec7e141a2fd9a0092d03f
Nov 03 21:30:14.350211 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:14.350193    4421 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:30:14.442071 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:14.442049    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fjmcb/must-gather-ltslt"]
Nov 03 21:30:14.444417 ip-10-0-143-48 kubenswrapper[4421]: W1103 21:30:14.444394    4421 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5eacb6e5_f8b4_4535_8888_b2049f8a5d1a.slice/crio-e08f0a2286aad983831ba2de84613d1b331c77e04aafc7be5720514981cfd863 WatchSource:0}: Error finding container e08f0a2286aad983831ba2de84613d1b331c77e04aafc7be5720514981cfd863: Status 404 returned error can't find the container with id e08f0a2286aad983831ba2de84613d1b331c77e04aafc7be5720514981cfd863
Nov 03 21:30:14.862218 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:14.862183    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x6xjk/must-gather-fr5j8" event={"ID":"03f212a0-dc19-4e3b-aa7c-88e97451e317","Type":"ContainerStarted","Data":"14f4b428ffc2f19ee659753c31b906cf7f8d04ab2e3ec7e141a2fd9a0092d03f"}
Nov 03 21:30:14.863244 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:14.863222    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fjmcb/must-gather-ltslt" event={"ID":"5eacb6e5-f8b4-4535-8888-b2049f8a5d1a","Type":"ContainerStarted","Data":"e08f0a2286aad983831ba2de84613d1b331c77e04aafc7be5720514981cfd863"}
Nov 03 21:30:15.873107 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:15.873066    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fjmcb/must-gather-ltslt" event={"ID":"5eacb6e5-f8b4-4535-8888-b2049f8a5d1a","Type":"ContainerStarted","Data":"119e0599bbbd17ef3ba99fcbcf34534eaf5f2de53e98eee585b2a60a427b2277"}
Nov 03 21:30:15.873535 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:15.873127    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fjmcb/must-gather-ltslt" event={"ID":"5eacb6e5-f8b4-4535-8888-b2049f8a5d1a","Type":"ContainerStarted","Data":"84ae6cfbc714a2bc6092bf04f809ba48101cd0a0cb8586145a1da92d1a6c2777"}
Nov 03 21:30:15.892988 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:15.892939    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fjmcb/must-gather-ltslt" podStartSLOduration=2.107320999 podStartE2EDuration="2.892924749s" totalImagesPullingTime="785.60375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:15.891482613 +0000 UTC m=+5346.024611075" watchObservedRunningTime="2026-11-03 21:30:15.892924749 +0000 UTC m=+5346.026053216"
Nov 03 21:30:17.259675 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:17.259642    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x8wrn_0181bad8-3459-4c64-8e31-12395fd4c6c7/global-pull-secret-syncer/0.log"
Nov 03 21:30:17.590283 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:17.590198    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-k5b5s_07e7cf29-701a-4c9d-a3ad-12d4846157c2/konnectivity-agent/0.log"
Nov 03 21:30:17.651418 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:17.651390    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-48.us-east-2.compute.internal_cc3d49b24d21c77c008662d39f9bb541/haproxy/0.log"
Nov 03 21:30:19.687962 ip-10-0-143-48 kubenswrapper[4421]: E1103 21:30:19.687879    4421 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-74j6n" containerName="extractor"
Nov 03 21:30:24.727051 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:24.727015    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cfb485dff-mjxfv_6f1f72d4-3cc6-4ae2-b9ac-a399a5b9a34c/authorino/0.log"
Nov 03 21:30:25.308015 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:25.307978    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jb45k_7e5c45c5-1f48-4b84-b7ae-244cd6e80b9f/kuadrant-console-plugin/0.log"
Nov 03 21:30:25.461321 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:25.461286    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-9p95v_5c786253-34d0-4b8b-8924-792d8cbc3d9f/manager/0.log"
Nov 03 21:30:27.470059 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:27.469987    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-k46fw_091b9d41-6f3f-4b38-b708-2ad312fc3af5/kube-state-metrics/0.log"
Nov 03 21:30:27.503501 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:27.503400    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-k46fw_091b9d41-6f3f-4b38-b708-2ad312fc3af5/kube-rbac-proxy-main/0.log"
Nov 03 21:30:27.534851 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:27.534784    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-k46fw_091b9d41-6f3f-4b38-b708-2ad312fc3af5/kube-rbac-proxy-self/0.log"
Nov 03 21:30:27.634197 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:27.634133    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8b6ng_2b03c6cd-cfd6-4234-820d-d42854548f2d/node-exporter/0.log"
Nov 03 21:30:27.660942 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:27.660900    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8b6ng_2b03c6cd-cfd6-4234-820d-d42854548f2d/kube-rbac-proxy/0.log"
Nov 03 21:30:27.686709 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:27.686616    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8b6ng_2b03c6cd-cfd6-4234-820d-d42854548f2d/init-textfile/0.log"
Nov 03 21:30:27.890859 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:27.890754    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-qsd4s_3971ef1f-1497-4eb4-b3a0-da2c5a75b619/kube-rbac-proxy-main/0.log"
Nov 03 21:30:27.921782 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:27.921734    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-qsd4s_3971ef1f-1497-4eb4-b3a0-da2c5a75b619/kube-rbac-proxy-self/0.log"
Nov 03 21:30:27.954005 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:27.953971    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-qsd4s_3971ef1f-1497-4eb4-b3a0-da2c5a75b619/openshift-state-metrics/0.log"
Nov 03 21:30:28.713057 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:28.711750    4421 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fjmcb/perf-node-gather-daemonset-txqtg"]
Nov 03 21:30:28.743835 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:28.743806    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fjmcb/perf-node-gather-daemonset-txqtg"]
Nov 03 21:30:28.744169 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:28.744153    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-txqtg"
Nov 03 21:30:28.853965 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:28.853930    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c050079c-be62-4384-956c-f9f416c09cc5-lib-modules\") pod \"perf-node-gather-daemonset-txqtg\" (UID: \"c050079c-be62-4384-956c-f9f416c09cc5\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-txqtg"
Nov 03 21:30:28.854168 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:28.853992    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4n52\" (UniqueName: \"kubernetes.io/projected/c050079c-be62-4384-956c-f9f416c09cc5-kube-api-access-w4n52\") pod \"perf-node-gather-daemonset-txqtg\" (UID: \"c050079c-be62-4384-956c-f9f416c09cc5\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-txqtg"
Nov 03 21:30:28.854221 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:28.854166    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c050079c-be62-4384-956c-f9f416c09cc5-sys\") pod \"perf-node-gather-daemonset-txqtg\" (UID: \"c050079c-be62-4384-956c-f9f416c09cc5\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-txqtg"
Nov 03 21:30:28.854361 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:28.854343    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c050079c-be62-4384-956c-f9f416c09cc5-proc\") pod \"perf-node-gather-daemonset-txqtg\" (UID: \"c050079c-be62-4384-956c-f9f416c09cc5\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-txqtg"
Nov 03 21:30:28.854412 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:28.854398    4421 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c050079c-be62-4384-956c-f9f416c09cc5-podres\") pod \"perf-node-gather-daemonset-txqtg\" (UID: \"c050079c-be62-4384-956c-f9f416c09cc5\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-txqtg"
Nov 03 21:30:28.955117 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:28.955078    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c050079c-be62-4384-956c-f9f416c09cc5-lib-modules\") pod \"perf-node-gather-daemonset-txqtg\" (UID: \"c050079c-be62-4384-956c-f9f416c09cc5\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-txqtg"
Nov 03 21:30:28.955327 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:28.955146    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4n52\" (UniqueName: \"kubernetes.io/projected/c050079c-be62-4384-956c-f9f416c09cc5-kube-api-access-w4n52\") pod \"perf-node-gather-daemonset-txqtg\" (UID: \"c050079c-be62-4384-956c-f9f416c09cc5\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-txqtg"
Nov 03 21:30:28.955327 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:28.955184    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c050079c-be62-4384-956c-f9f416c09cc5-sys\") pod \"perf-node-gather-daemonset-txqtg\" (UID: \"c050079c-be62-4384-956c-f9f416c09cc5\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-txqtg"
Nov 03 21:30:28.955327 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:28.955198    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c050079c-be62-4384-956c-f9f416c09cc5-lib-modules\") pod \"perf-node-gather-daemonset-txqtg\" (UID: \"c050079c-be62-4384-956c-f9f416c09cc5\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-txqtg"
Nov 03 21:30:28.955327 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:28.955265    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c050079c-be62-4384-956c-f9f416c09cc5-proc\") pod \"perf-node-gather-daemonset-txqtg\" (UID: \"c050079c-be62-4384-956c-f9f416c09cc5\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-txqtg"
Nov 03 21:30:28.955327 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:28.955292    4421 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c050079c-be62-4384-956c-f9f416c09cc5-podres\") pod \"perf-node-gather-daemonset-txqtg\" (UID: \"c050079c-be62-4384-956c-f9f416c09cc5\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-txqtg"
Nov 03 21:30:28.955703 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:28.955401    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c050079c-be62-4384-956c-f9f416c09cc5-podres\") pod \"perf-node-gather-daemonset-txqtg\" (UID: \"c050079c-be62-4384-956c-f9f416c09cc5\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-txqtg"
Nov 03 21:30:28.955703 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:28.955457    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c050079c-be62-4384-956c-f9f416c09cc5-sys\") pod \"perf-node-gather-daemonset-txqtg\" (UID: \"c050079c-be62-4384-956c-f9f416c09cc5\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-txqtg"
Nov 03 21:30:28.955703 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:28.955504    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c050079c-be62-4384-956c-f9f416c09cc5-proc\") pod \"perf-node-gather-daemonset-txqtg\" (UID: \"c050079c-be62-4384-956c-f9f416c09cc5\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-txqtg"
Nov 03 21:30:28.971301 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:28.971198    4421 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4n52\" (UniqueName: \"kubernetes.io/projected/c050079c-be62-4384-956c-f9f416c09cc5-kube-api-access-w4n52\") pod \"perf-node-gather-daemonset-txqtg\" (UID: \"c050079c-be62-4384-956c-f9f416c09cc5\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-txqtg"
Nov 03 21:30:29.092493 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:29.092241    4421 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-txqtg"
Nov 03 21:30:29.417748 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:29.417660    4421 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fjmcb/perf-node-gather-daemonset-txqtg"]
Nov 03 21:30:30.123625 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:30.120819    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-txqtg" event={"ID":"c050079c-be62-4384-956c-f9f416c09cc5","Type":"ContainerStarted","Data":"c2ca7c5bc757a35f2447985f34f3e34769292049df8d729fad4d27261e42925c"}
Nov 03 21:30:30.123625 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:30.120885    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-txqtg" event={"ID":"c050079c-be62-4384-956c-f9f416c09cc5","Type":"ContainerStarted","Data":"479e3775209c5132167507908728bc59302c59e894343d168bfee12d6df435b3"}
Nov 03 21:30:30.123625 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:30.121723    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-txqtg"
Nov 03 21:30:30.142162 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:30.142107    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-txqtg" podStartSLOduration=2.142088728 podStartE2EDuration="2.142088728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:30.140557844 +0000 UTC m=+5360.273686306" watchObservedRunningTime="2026-11-03 21:30:30.142088728 +0000 UTC m=+5360.275217191"
Nov 03 21:30:31.503034 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:31.503004    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8596678df6-sgrph_14b0cc21-b4a2-4591-854e-91a62ab296c0/console/0.log"
Nov 03 21:30:33.160698 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:33.156309    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x6xjk/must-gather-fr5j8" event={"ID":"03f212a0-dc19-4e3b-aa7c-88e97451e317","Type":"ContainerStarted","Data":"c1c1e7b2d6eecb55a6f9238c5add7a58b0f2b64b7e058ed69662738b46e8293b"}
Nov 03 21:30:33.160698 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:33.156363    4421 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x6xjk/must-gather-fr5j8" event={"ID":"03f212a0-dc19-4e3b-aa7c-88e97451e317","Type":"ContainerStarted","Data":"871be4ef007f2d1fa91506cb55b03d39bedfd95a1c5763f7da8e866ebf55812d"}
Nov 03 21:30:33.178039 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:33.177988    4421 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-x6xjk/must-gather-fr5j8" podStartSLOduration=1.925880867 podStartE2EDuration="20.177973807s" totalImagesPullingTime="18.25209294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:33.176964404 +0000 UTC m=+5363.310092866" watchObservedRunningTime="2026-11-03 21:30:33.177973807 +0000 UTC m=+5363.311102269"
Nov 03 21:30:33.814305 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:33.814260    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wb9sn_eaaa35b2-aee2-40f8-956b-0b9f697dc994/dns/0.log"
Nov 03 21:30:33.838830 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:33.838781    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wb9sn_eaaa35b2-aee2-40f8-956b-0b9f697dc994/kube-rbac-proxy/0.log"
Nov 03 21:30:33.869960 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:33.869927    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g2rp5_153d1ce3-22c2-444c-9bd2-ee91ff044d29/dns-node-resolver/0.log"
Nov 03 21:30:34.327990 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:34.327960    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-2725f_1c4bb3bf-caf2-4932-8b65-00189e8d41fb/manager/0.log"
Nov 03 21:30:34.610076 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:34.610006    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7679cb5584-qgtd8_8953fd4e-dc26-4422-82bb-1c2e6fe569b5/registry/0.log"
Nov 03 21:30:34.703503 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:34.703457    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tt4df_78003043-36f1-452a-8ccf-37b574877715/node-ca/0.log"
Nov 03 21:30:36.218034 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:36.217999    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-687b69bd9-hmtjv_670212d6-aae1-4981-aaf1-5b12df51bc0d/manager/0.log"
Nov 03 21:30:36.229986 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:36.229962    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-8lzxb_c23a53b9-ca79-461a-a8f6-20618ffc35aa/server/0.log"
Nov 03 21:30:37.155939 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:37.155909    4421 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-txqtg"
Nov 03 21:30:38.134123 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:38.134087    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-99ntv_b347c3ff-a128-4fd2-8c07-dfea5c60a431/manager/0.log"
Nov 03 21:30:38.155217 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:38.155189    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-4bbvd_1dcbd7f2-5145-444c-9b8f-7035eb14b64e/proxy/0.log"
Nov 03 21:30:39.483891 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:39.483855    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-2725f_1c4bb3bf-caf2-4932-8b65-00189e8d41fb/manager/0.log"
Nov 03 21:30:42.270680 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:42.270637    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-687b69bd9-hmtjv_670212d6-aae1-4981-aaf1-5b12df51bc0d/manager/0.log"
Nov 03 21:30:42.290882 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:42.290857    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-8lzxb_c23a53b9-ca79-461a-a8f6-20618ffc35aa/server/0.log"
Nov 03 21:30:43.249283 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:43.249237    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-594f764c95-z8fjv_b5520324-5b0c-4417-8221-a2cf1c08b881/kube-auth-proxy/0.log"
Nov 03 21:30:43.295322 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:43.295287    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-6bhmr_9d6e2ee7-7d48-4879-b6fa-f303b5d24697/istio-proxy/0.log"
Nov 03 21:30:43.364629 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:43.364561    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-bnt6k_9375775d-7b1d-48d0-ab7b-ce8cf03729e9/istio-proxy/0.log"
Nov 03 21:30:43.406367 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:43.406335    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-cd5ll_b2a7c2ad-1827-4aa9-a327-aa1ac8bc2b0d/istio-proxy/0.log"
Nov 03 21:30:43.610913 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:43.610820    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-xfwgr_bc25e3fb-91ba-4d75-92d3-d1aab2feb033/istio-proxy/0.log"
Nov 03 21:30:43.704305 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:43.704260    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-5556d898cf-4gn6l_835ed6d1-8b5c-4b49-8316-0fbb1d161e2c/payload-pre-processing/0.log"
Nov 03 21:30:43.736198 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:43.736167    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7dc89df4c4-hhqqf_e78b2b21-5706-465b-8ea1-9f1679e0f4ff/payload-processing/0.log"
Nov 03 21:30:44.152471 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:44.152413    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-99ntv_b347c3ff-a128-4fd2-8c07-dfea5c60a431/manager/0.log"
Nov 03 21:30:44.227087 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:44.227056    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-4bbvd_1dcbd7f2-5145-444c-9b8f-7035eb14b64e/proxy/0.log"
Nov 03 21:30:45.333047 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:45.333017    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-2725f_1c4bb3bf-caf2-4932-8b65-00189e8d41fb/manager/0.log"
Nov 03 21:30:46.666666 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:46.666636    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sx9rt_126ae513-97e3-4ef0-b415-5ec03ea09960/serve-healthcheck-canary/0.log"
Nov 03 21:30:47.346255 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:47.346217    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-74j6n_20b2d254-c5b2-4a59-8ce9-49a9f2c0f435/kube-rbac-proxy/0.log"
Nov 03 21:30:47.374428 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:47.374375    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-74j6n_20b2d254-c5b2-4a59-8ce9-49a9f2c0f435/exporter/0.log"
Nov 03 21:30:47.406162 ip-10-0-143-48 kubenswrapper[4421]: I1103 21:30:47.406102    4421 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-74j6n_20b2d254-c5b2-4a59-8ce9-49a9f2c0f435/extractor/0.log"